xmlrpc posting with Xpostulate

[tonybaldwin]

Okay...I've started to hack up the xmlrpc functions.
First, I had to replace all "<" and ">" in html tags in a post with"&lt;" and "&gt;",
and then, I have basically just handwritten out a relevant xml file.
Which looks like this:

<?xml version="1.0"?><methodCall><methodName>LJ.XMLRPC.postevent</methodName><params><param><value><struct><member><name>username</name><value><string>tonytraductor</string></value></member><member><name>password</name><value><string>XXXXXXXX</string></value></member><member><name>subject</name><value><string>xmlrpc posting</string></value></member><member><name>lineendings</name><value><string>pc</string><value></member><member><name>year</name><value><int>2010</int></value></member><member><name>mon</name><value><int>04</int></value></member><member><name>day</name><value><int>06</int></value></member><member><name>hour</name><value><int>10</int></value></member><member><name>min</name><value><int>46</int></value></member><member><name>prop_current_location</name><value><string>127.0.0.1</string></value></member><member><name>propr_current_mood</name><value><string>focused</string></value></member><member><name>prop_taglist</name><value><string>xmlrpc, xpostulate, wordpress, livejournal, dreamwidth, insanejournal</string></value></member><member><name>prop_currentmusic</name><value><string>none</string></value></member><member><name>userjournal</name><value><string>which journal?</string></value></member><member><name>event</name><value><string>Okay...I've started to hack up the xmlrpc functions.
First, I had to replace all "<" and ">" in html tags in a post with "&lt;" and "&gt;",
and then, I have basically just handwritten out a relevant xml file.


</string></value></member></struct></value></param></params></methodCall>


What I haven't done is tried to send such a file to a server, yet...
I'm not sure if the posting method has to be different from a flat post, in terms of simply sending this file as the post data...
More work...
But, hey, I'm working on it!

Comments

[catness]

Aren't there any low-level libraries in Tcl which convert array to xml-rpc data? Then you can use it in a more straightforward way, as xml-rpc is supposed to add convenience, not pain in the ass ;) For example, in C++ it goes something like that:

....params is an array (hash).......
params["username"] = username;
params["password"] = password;
....
.......props - another array......
props["current_mood"] = mood;
props["current_music"] = music;
params["props"] = props;
.......
and then just use the function which converts the array to proper xml-rpc data and sends to the server.

[tonybaldwin]

There is a Tcl xmlrpc module.
I read it.
I'm not sure how to use it, oddly.
I kind of see how it builds the "request", or, basically, the xml file, which I created (although differently), but I don't understand how it is sending the request.
I've tried several ways to send it with Tcl's http, which is how I sent flat posts, but it's not working for me.
I'm probably going to have to figure out this existing module.
I didn't want to use an external module, mostly to simplify installation and use for clueless users, really.
But, I suppose if I get it implemented and working, I can then make a starkit, which is basically a standalone binary, which will run on any platform, thus making installation and use as simple as possible.
I'm probably going to go that route.
Oh, but wait...the thing is, I am having difficulty running the existing xmlrpc module, because it requires stuff from ActiveTcl, which is not included in debian's repos.
I really don't want to remove the debian tcl and install ActiveState's distribution, which I'd have to do to get this module working.
Then, I think, if this is a headache for me to get running, how is it going to work for the average user?

[tonybaldwin]

I haven't had any trouble creating the xml data to send, I'm just not succeeding in sending it correctly.
Apparently, it's quite different than sending data à la flat posting method.
I might have better luck with cURL, than with Tcl's http, but I'm trying to use Tcl resources, again, to simplify installation and use for everybody (including windows users...yech).

[catness]

Maybe your headers are wrong? make sure it's Content-Type: text/xml, Content-Length is correct, and the url is /interface/xmlrpc.

Yeah, it doesn't make sense to require the user to install a crapload of stuff. (That's why btw I can't stand Python and Ruby applications - on my system, most of them require upgrades and addons which are incompatible with the current versions, and something inevitably breaks)

[tonybaldwin]

Oh yeah, I've read the CODE, but it doesn't come with any documentation I can make sense of.
The code is commented, but, still, can't figure out how it's sending the request.
Figured I should clarify that.

[tonybaldwin]

The thing that sucks is, I keep getting "ok" back from the server, but the post doesn't appear on the journal.

[tonybaldwin]

oh...that doesn't show the xml tags, of course.
but, I have the param and props and post all in the same [param][params][value][struct] [/struct][/params][/param]
Do they need to be separate?
ie;
headers here
[?xml version="1.0"?]
[methodCall][methodName>LJ.XMLRPC.postevent[/methodName]
[param]
[params]
[member][name]username[/name][value][string]tonytraductor[/string][/value][/member]
etc.
[/params]
[/param]
[param]
[params]
[member][name]prop_currentmood[/name][string]annoyed[/string][/value][member][/params][/param]
[param][params][member][name]event[/name][value][string]some text here...
this will be the post...yippee...
[/string][/value][/member]
[/param]
[/params]
[/methodCall]

[catness]

yes, params and props are different. Sorry I skipped over the xml data in your post without verifying if it's correct. here's a properly formed example (dumped my data stream while sending) :
<?xml version="1.0"?>
<methodCall>
<methodName>LJ.XMLRPC.postevent</methodName>
<params>
<param>
<value><struct>
<member>
<name>auth_challenge</name>
<value><string>blablabla</string></value>
</member>
<member>
<name>auth_method</name>
<value><string>challenge</string></value>
</member>
<member>
<name>auth_response</name>
<value><string>blablabla</string></value>
</member>
<member>
<name>day</name>
<value><int>6</int></value>
</member>
<member>
<name>event</name>
<value><base64>aGVsbG8gd29ybGQKLS0tLS0KaGVsbG8KLS0tLS0=</base64></value>
</member>
<member>
<name>hour</name>
<value><int>23</int></value>
</member>
<member>
<name>lineendings</name>
<value><string>unix</string></value>
</member>
<member>
<name>min</name>
<value><int>9</int></value>
</member>
<member>
<name>mode</name>
<value><string>postevent</string></value>
</member>
<member>
<name>mon</name>
<value><int>4</int></value>
</member>
<member>
<name>props</name>
<value><struct>
<member>
<name>current_mood</name>
<value><string>awake</string></value>
</member>
<member>
<name>current_moodid</name>
<value><int>87</int></value>
</member>
<member>
<name>opt_backdated</name>
<value><int>0</int></value>
</member>
<member>
<name>taglist</name>
<value><string>hello, world</string></value>
</member>
<member>
<name>useragent</name>
<value><string>QTXPost/0.3.0</string></value>
</member>
</struct></value>
</member>
<member>
<name>security</name>
<value><string>public</string></value>
</member>
<member>
<name>subject</name>
<value><string>test</string></value>
</member>
<member>
<name>username</name>
<value><string>my_username</string></value>
</member>
<member>
<name>ver</name>
<value><int>1</int></value>
</member>
<member>
<name>year</name>
<value><int>2010</int></value>
</member>
</struct></value>
</param>
</params>
</methodCall>

[tonybaldwin]

This is the kind of thing I'm not finding sufficiently explained in the existing documentation.
I hate that.
How'd you post with the < in there?
I seem to be missing some stuff, too.
(auth_challenge, auth_method, auth_respons...don't know what needs to be in there (sure it's not "blablabla", too), haven't seen it)
So, clearly, I have some more learning to do, but, the thing is, the documentation I have found is woefully inadequate in explaining this things...at least for a rather novice programmer, such as myself.

[catness]

Never mind auth_challenge, I suppose it should work with plain passwords for a start - but have a look at http://www.livejournal.com/doc/server/ljp.csp.xml-rpc.getchallenge.html (I can explain further if you need)
(and I've replaced the real data with blablabla just in case, for security - the challenge is supposed to expire after posting, but I'm a bit paranoid)

Also, probably you don't have to base64-encode the post content, unless you post in non-English languages with weird characters.

For using angle brackets, replace them with &gt; and &lt; ;)

The LJ documentation is a bit terse, that's right... but you'll get the hang of it eventually!

curioser and curioser

[tonybaldwin]

Okay, I'm trying to use tdom (tcl parser) to get the challenge var.
I'm doing:

set getchal "<?xml version=\"1.0\"?>
<methodCall>
<methodName>LJ.XMLRPC.getchallenge</methodName>
<params>
<param>
<value><struct>
</struct></value>
</param>
</params>
</methodCall>"

set getchallenge [http::geturl http://www.livejournal.com/interface/xmlrpc -query $getchal -type "text/xml" ]
set mychal [http::status $getchallenge]
set doc [dom parse $mychal]
set root [$mychal documentElement]
set nodeList [$root selectNodes {descendant::a}]
set chalvar {}

foreach node $nodeList {
set attList [$node attributes *]
foreach attribute $attList {
if {[string tolower $attribute] == "c0:"} {
set mychal [$node getAttribute $attribute]
break
}
}
}


But apparently http::meta or http::status is NOT the response from the server that needs to be parsed to grab me the infor (challenge variable c0:blablablab) that I need.
Because tdom keep throwing an error, suggesting that what I'm feeding (which, I thought was the server response) it is not valid xml.

Re: curioser and curioser

[tonybaldwin]

wait...
does $getchal actually need something in the params/param/value/struct?
Because, I just copied from your post http://catness.dreamwidth.org/61674.html
Do I need to include username/password or something in that struct?
let me try that...maybe I'll get a valid response.
Although, I think I'm just feeding tdom the wrong thing...not sure.

Re: curioser and curioser

[tonybaldwin]

nope...still keep getting an error from the parser
syntax error on line 1 col 0

Re: curioser and curioser

[tonybaldwin]

I'm up on #tcl @freenode trying to sort this out

Re: curioser and curioser

[catness]

No, getchallenge does NOT need username/password (that's why it's secure ;) You can see exactly what it needs:
http://www.livejournal.com/doc/server/ljp.csp.xml-rpc.getchallenge.html

re:base64, cyrillic characters certainly require encoding, not sure about European languages, I think German characters worked without it...

Re: curioser and curioser

[tonybaldwin]

well then, if ß works, ç should be fine.

Content-Length: 97
this should basically be static, right?
I don't need to fish that var out of anywhere, since, it appears the very same text/xml is sent for every challenge.

Re: curioser and curioser

[catness]

Can you print out the full response from the server (before parsing it)?

[tonybaldwin]

I got the challenge!
I got it!

Oddly, my post still doesn't appear...
I have

<?xml version="1.0"?>
<methodCall><methodName>LJ.XMLRPC.postevent</methodName>
<params><param><value><struct>
<member><name>auth_scheme</name><value><string>c0</string></value></member>
<member><name>auth_method</name><value><string>clear</string></value></member>
<member><name>challenge</name><value><string><?xml version="1.0" encoding="UTF-8"?><methodResponse><params><param><value><struct><member><name>auth_scheme</name><value><string>c0</string></value></member><member><name>server_time</name><value><int>1270660801</int></value></member><member><name>challenge</name><value><string>c0:1270659600:1201:60:ia856BGhJT9RoL1XgxbN:edfe983180681a5ec49686501358dab6</string></value></member><member><name>expire_time</name><value><int>1270660861</int></value></member></struct></value></param></params></methodResponse></string></value></member>
etc., etc.

sending it with
set dopost [http::geturl http://www.livejournal.com/interface/xmlrpc -query $post -type "text/xml" ]

Frank says "ok"
but no post appears on my LJ...
I get no errors.

Making progress! but, something still needs work, obviously...
off to NYC now...
back later

[catness]

Yay!

Probably you're not creating the response correctly - it's not just sending the challenge back to the server, it's creating an md5 hash with it and password.

[catness]

Meanwhile, I dunno about the challenge but I've succeeded to post with cleartext password just fine, with the following script (though it's my first attempt on using tcl so I don't really know what I'm doing, lol). You can see the result at http://catness.insanejournal.com/204215.html
(the server returns the url among the result data)

#!/usr/bin/tclsh

package require http;

set url "http://www.insanejournal.com/interface/xmlrpc"

set password "MYPASSWORD"
set username "catness"

set post "<?xml version=\"1.0\"?>
<methodCall>
<methodName>LJ.XMLRPC.postevent</methodName>
<params>
<param>
<value><struct>
<member>
<name>day</name>
<value><int>7</int></value>
</member>
<member>
<name>event</name>
<value><string>This is a test. Please ignore.</string></value>
</member>
<member>
<name>hour</name>
<value><int>21</int></value>
</member>
<member>
<name>lineendings</name>
<value><string>unix</string></value>
</member>
<member>
<name>min</name>
<value><int>40</int></value>
</member>
<member>
<name>mon</name>
<value><int>4</int></value>
</member>
<member>
<name>props</name>
<value><struct>
<member>
<name>current_mood</name>
<value><string>awake</string></value>
</member>
<member>
<name>current_moodid</name>
<value><int>87</int></value>
</member>
<member>
<name>taglist</name>
<value><string>test</string></value>
</member>
<member>
<name>useragent</name>
<value><string>tcl</string></value>
</member>
</struct></value>
</member>
<member>
<name>security</name>
<value><string>public</string></value>
</member>
<member>
<name>subject</name>
<value><string>test, please ignore</string></value>
</member>
<member>
<name>username</name>
<value><string>$username</string></value>
</member>
<member>
<name>password</name>
<value><string>$password</string></value>
</member>
<member>
<name>ver</name>
<value><int>1</int></value>
</member>
<member>
<name>year</name>
<value><int>2010</int></value>
</member>
</struct></value>
</param>
</params>
</methodCall>"

puts $post

puts $url

set post_response [http::geturl $url -query $post -type "text/xml" ]

puts [http::data $post_response]

[tonybaldwin]

Hmmm...
i'm getting this in the http::data response:
"Can't call method "is_validated" on an undefined value at /home/lj/cgi-bin/Apache/Livejournal.pm line 1814"

wonder what I'm still doing wrong...or why LJ is being a b1+ch when IJ works, or something...(I haven't tried IJ, but I'm doing nearly precisely what you did with IJ, only to LJ, with no joy)...

[catness]

And I fixed the challenge too (got curious - damn you ;) Here's the code which creates $response from $challenge and $password - note the "tolower" bit, cost me a couple of hours, doesn't work without it... all the rest is the same as in the previous example, except for using auth_challenge and auth_response, of course. The whole script is too long to post because it's using some xml processing function which I've found elsewhere, but I can send it if you need.

set encoded [string tolower [md5::md5 -hex $password] ]
set cat $challenge$encoded
set response [string tolower [md5::md5 -hex $cat] ]

This post starts to be difficult to navigate... ;)

success, at last!

[tonybaldwin]

http://tonytraductor.livejournal.com/291471.html

I DID IT!!!
FINALLY!!!

Re: success, at last!

[tonybaldwin]

this worked:

################
# post to livejournal...

proc ljpost {} {

# changing html tags for insertion in xml
set content [.txt.txt get 1.0 end]
set escaped [string map {
"<" "<"
">" ">"
\" "\""
} $content]
.txt.txt delete 1.0 end
.txt.txt insert insert $escaped
# is it redundant that ptext and escaped from the above are the same?
# probably
set ptext [.txt.txt get 1.0 {end -1c}]

global mypost
set mypost "<?xml version=\"1.0\"?>
<methodCall><methodName>LJ.XMLRPC.postevent</methodName>
<params><param>
<value><struct>
<member><name>year</name><value><int>$::year</int></value></member>
<member><name>mon</name><value><int>$::mon</int></value></member>
<member><name>day</name><value><int>$::day</int></value></member>
<member><name>hour</name><value><int>$::hour</int></value></member>
<member><name>min</name><value><int>$::min</int></value></member>
<member><name>usejournal</name><value><string>$::usej</string></value></member>
<member><name>event</name><value><string>$ptext</string></value></member>
<member><name>username</name><value><string>$::ljname</string></value></member>
<member><name>password</name><value><string>$::ljpswd</string></value></member>
<member><name>subject</name><value><string>$::subject</string></value></member>
<member><name>lineendings</name><value><string>unix</string></value></member>
<member><name>security</name><value><string>$::priv</string></value></member>
<member><name>ver</name><value><int>1</int></value></member>
<member><name>props</name>
<value><struct>
<member><name>useragent</name><value><string>Xpostulate</string></value></member>
<member><name>current_location</name><value><string>$::loc</string></value></member>
<member><name>current_mood</name><value><string>$::mood</string></value></member>
<member><name>taglist</name><value><string>$::tags</string></value></member>
<member><name>current_music</name><value><string>$::tunes</string></value></member>
</struct></value></member>
</struct></value>
</param></params>
</methodCall>"

global plength
set plength [string length $mypost]

set dopost [http::geturl http://www.livejournal.com/interface/xmlrpc -query $::mypost -type "text/xml" ]
set ljmta [http::meta $dopost]
set ljstat [http::status $dopost]
set ljresponse [http::data $dopost]
# upvar #0 $dopost state
# puts $state(body)

toplevel .rsp
wm title .rsp "Post Status"

frame .rsp.btns
grid [tk::label .rsp.btns.lbl -text "Frank says: $ljstat\nPost length: $::plength"]
grid [tk::button .rsp.btns.view -text "View Journal" -command {
set ljv "http://$::usej.livejournal.com"
exec $::brow $ljv &
}]\
[tk::button .rsp.btns.ok -text "DONE" -command {destroy .rsp}]

frame .rsp.txt
text .rsp.txt.t -width 80 -height 20
.rsp.txt.t insert end $ljresponse

pack .rsp.btns -in .rsp -side top -fill x
pack .rsp.txt.t -in .rsp.txt -side top -fill x
pack .rsp.txt -in .rsp -side top -fill x

}

It seems to be working on LJ, IJ, and DW.
Iziblog apparently doesn't play with xmlrpc (but flatposting works fine there), and, I don't have a deadjournal to test that.

Now, I just have to look at the difference between lj.xmlrpc and wordpress, and I'll have wordpress posting working!
The google api for blogger can't be too far a stretch, either...

Re: success, at last!

[catness]

Wh00t!!!! :D :D

I suggest you should add challenge/response at some stage - sending password in clear text over an unencrypted connection makes a lot of people twitchy.

Also if you want more LJ-clones for testing, I've been using http://www.inksome.com and http://www.scribbld.net/ . (I used to have DJ but deleted it; as far as I remember, it did work with XML-RPC)