OpenAniDB: Reinventing the Wheel

Hey, guys. In a lot of ways, the code I've been writing is fundamentally different than Python, and I keep finding myself writing code to do things that wx can do natively. (Would you believe I reinvented the hash map for strings before learning about wxHashMap?)

So, the state of things. UDP is pretty solid. It currently blocks on I/O. Ideally, we would select() and re-enter the code path once we get a response, but this is pretty good considering we are single-threaded. Also, instead of small nonce functions for each response/error code, I'm now letting each part of the app just switch{} between each code, and then the UDP object cleans up and handles any errors received. I'm sure there will be more tweaking, but I think it's pretty cool to be able to just "have things work" in regards to network errors. For example, if you receive a 555 ("Banned!") at any point during execution, the UDP object will pop up a box letting the user know, without bothering the main code.

The DB is, well, the DB. There's a lot of things unimplemented that the old DB system had; in particular, I have ditched the old per-table functions for a nice, generic Put() and Get(), which handle raw SQL statements. There's also PutRaw() and GetOne(), which wrap those into functions that are much more palatable. There's still a lot of app code which is missing. I haven't rebuilt the morass of code that drives the DB browser. I'm looking forward to it, though. It will be quite cool, I hope.

Hashing works. So does adding to the mylist. Hell, if that's all you really want, then it's done. Finished. The ability to load files into the hasher, hash 'em, and then send the hashes to AniDB is all there. However, renaming/moving hasn't been reimplemented, so if you need that, you're out of luck. Also, the whole thing with DB caching isn't quite working (and there's no way to test it without the DB browser,) so more code must be written.

Good news, though. Hashing is fast. I'm using a builtin MD4 instead of external libs, and frankly, it's ridiculously fast. 10-15 seconds to hash 170 MB. On Linux, at least, I can cache the entire file in memory after hashing, and then just do additional hashes from RAM, which is impressively speedy. On Windows, eh, not so much.

~C.

Traditional - "The Christmas Song"

Happy holidays! ~C.

Music: Corbin Simpson - "Quick Adventure"

One of my adventures. This recording was a warmup before my jazz audition. (He stopped me halfway through the audition, said he had had enough and I was the guy. I'm the guy. Oh yeah. *dances*) ~ C.

Quirk

Something interesting I found: wxConfigBase::Read() is improperly overloaded. GCC accepts:

bool wxConfigBase::Read(wxString, true);

...but not...

bool wxConfigBase::Read(wxString, false);

Very interesting, isn't it? My current hypothesis is that false is internally represented by 0, which is the null pointer, so it isn't sure if it should be (bool)false or (bool*)NULL.
My solution:

bool f = false;
bool wxConfigBase::Read(wxString, f);

Inelegant, but it works. Defaults for settings work. Next on the list is file renaming.
~ C.

Music: Portal - "Still Alive"

The infamous radio song from Portal. Cross-posted on Youtube. More tomorrow. ~C.

OpenAniDB: Settings

Settings work. Yay. I'm actually rather fond of the setup as it stands, because the settings calls wxConfigBase::Write() and wxConfigBase::Read() are overloaded for bools, longs, doubles, floats, and strings. Very cool.

I can now hash a file from within the GUI and add it to the mylist. Now that I'm finished with school for the fall, I will probably be putting in a bit more and hopefully will be ready to commit to Subversion in a few weeks or perhaps sooner. The speed of C++ cannot be underestimated: I am hashing files in less than ten seconds now!

~ C.

Living Dead Week

Pardon the title pun, but I thought it was necessary considering what I've got for all of you. ("You," I know, is only like three people, but still...)

First off, OpenAniDB has a local DB again. db.py has been distilled from fifteen-odd functions into a singleton with three member functions. I'm pretty happy with it. Right now I'm at ~750 lines of code; if I end up with 2,000 I'll be a happy (and impressed, if I do say so myself) Corbin.

Second, I got my webcam working! It's a tiny bit slow, but does the job alright, so I recorded a sketch of Footprints in celebration. Here you go:


See you all later.
~ C.

OpenAniDB: Dead Week

So, first things first: Justin and I have reached a very real roadblock on OpenSmash; we need artwork to proceed. We're looking into a couple sources, but for now it's been placed on the back burner.

I got hashing working. Lemme talk about hashes. The hash primarily used by AniDB is called ED2K, after the filesharing program (eDonkey 2000) that it was designed for. The hash is pretty simple; it's a tree hash using MD4 and one layer of leaves. My algorithm is pretty similar to the original.

1. Read in up to 9.5 MiB, and hash it using MD4.
2. Once end-of-file is reached, concatenate the hashes of the file pieces, and hash it using MD4.

My bright idea was to use a static 9.5 MiB buffer on the stack to hold the piece currently being hashed; that way, there would be no memory allocation problems. However, as soon as I declared it, I started getting segmentation faults. Turns out that a buffer of that size, on the stack, forces that entire scope to be on a different page of memory, so all of the pointers back into the application caused segfaults. Needless to say, that large buffer is now dynamically allocated at the start of hashing and released at the end. Still, that puts our memory usage at about 80MB less than AOM, so I'm okay with it.

SQLite internals are getting worked on, bit by bit. The API is completely different from anything else I've ever used, including APSW. Massive amounts of callbacks. It feels like Windows API programming.

Have fun with finals, guys!
~ C.

OpenAniDB: Sick Day

Today was a sick day, which is bad for my classes since I'm too sore to make music, but it was a good day for code. Today I was able to log in, make a few requests, and then close the connection, all without using anything besides the C++ GUI. I'm almost up to speed on the C++ side of wx, although I keep discovering new and strange uses for wxString.

On the OpenSmash side, we're gonna work during Thanksgiving weekend; we've got an entire day set aside for it. Have a good turkey (or tofurkey) day, everybody!

Edit: We are now Unicode-compliant. There shouldn't be any problems with kana this time around. I need sleep now.

~ C.

OpenAniDB: Settings Dialog

Sorry 'bout the lack of posts this week. Real-world issues.

Q&A time! Asks jaroslav:

Are you training implement same rename system like in WebAOM? It would be good, wouldn't it? Or is already implement?
And which system of local database will you use? Same like in AOM or WebAOM or different?

Well, I'll probably reimplement the simple rename system I did before. After that, I may do the WebAOM system if people need that kind of advanced templating. As for the local database, it's SQLite. AOM also uses SQLite, but their database files are encrypted and I do not have the keys needed to decrypt those.

Anyway, as for status, I've got the settings dialog working again, except for that whole thing where settings need to be persistent. I'll keep plugging away. Tomorrow is the installfest for the local LUG, so I may or may not work depending on business.

~ C.

OpenAniDB: UDP Works

And that's all I really have to say about that. (Click to enlarge.)

~ C.

OpenAniDB: C++ Rewrite Status

Figured since it's been five whole days (gasp!) I should let you guys in on the status of things. The C++ rewrite was something I had on the back burner as a way to enforce some coding habits, like type safety and thread safety, missing in Python. It's still in wx, but should be much faster and cleaner. Anyway, here's what's up.

Files:

• openanidb.py: Complete. This file was generated by openanidb.wxg and wxGlade, and its contents have not really changed. Well, the guts of the events are different, but the structure is the same. This file draws the actual window and such.
• config.py: Not started. As far as I know, there's no wx equivalent for the settings/config file system in Python. As you might imagine, I am not looking forward to having to write my own.
  
• db.py: Not started. It's pretty much the same SQLite antics as before. I haven't decided whether or not to put it in a thread or just reentrant functions, but it's not going to be very difficult.
  
• hash.py: Not started. This will probably be the last file to be ported. It's like two inline functions for MD4/eMule hashing.
  
• setup.py: N/A. Not going to be ported. I have a Makefile instead.
  
• udp.py: Incomplete. The thread structure is filled out, mostly. Still needs hooks for DB and config, which are (unsurprisingly) not started. Also need UDP code, which is missing from wx 2.6. It's in wx 2.8, but neither Debian nor Gentoo have it. I'm gonna kill something. In the meantime, I need to find some way around it; probably some kind of supplementary UDP code.
  

Isn't that all so very pretty? So, don't think I'm not working. Or, say I'm not working enough. Whichever.
~C.

OpenSmash: SVN r4

Well, I finished the DirectInput joystick handler, which means that Win32 joysticks should work. They work in MinGW, at least. There's still a couple buggies, but those are most likely due to Wine and not to my code, since they're in the Direct3D section and the joysticks are in DirectInput. Nasty details follow.

So, on Win32, a set of helper functions are needed in order to track DirectInput. JoystickManager has to track the DirectInput8 interface, and each Joystick has to track a DirectInputDevice8. It's fucking ridiculous. I bet if I counted lines, there would be more DirectInput code than Linux code. Hmm. Lemme check.

$ i686-pc-linux-gnu-g++ -E -I/usr/include/irrlicht joystick.cpp | wc -lL -
58901 235 -
$ mingw32-g++ -E -I/usr/mingw32/usr/include/irrlicht joystick.cpp | wc -lL -
68855 529 -

Wow. The MinGW version, with headers, is almost 10,000 lines more of code. The second number is "number of characters in longest line," which is another staggering comparison: 529 vs. 235 characters.

"But wait," you say, "in your sleep-deprived excitement you forgot that the headers for different compilers are different sizes!"

"Ah, yes," says I, "I did not forget. Let us run the comparison again."

Manually checking the preprocessor output, I find that the Linux version's Joystick class def is at line 58719, while the Win32 version's is at 68675. Thus, the Linux joystick.h/cpp is 182 lines, and Win32 is 180. Too close to call, really. The other file, joystickmanager.h/cpp, is only 30 lines more expensive in Windows than in Linux. (But 30/180 is 1/6, so you could say that it's a 20% increase from Linux to Windows. Ah, the joys of statistics.)

Looks like my hypothesis was wrong. Nonetheless, having to enumerate and calibrate axes manually is a pain in the ass. If you want shits 'n' giggles, take a look at the source code, especially joystick.cpp. Tons of fun in there.

Summary:

• DirectInput handler for joysticks.
• Official Windows 32-bit support.
• Subdivision of Joystick and JoystickManager classes into Windows and Linux parts.
• Undying hate of Microsoft programming interfaces reaffirmed and vindicated.
  

Also, I haven't actually tried building this on Win32. That's Justin's task. I think. I hope.

See you later.
~C.

OpenSmash: SVN r3

Well, revision 3 has finally been committed. I've been a busy beaver with music for my classes, but I decided to crack down and get MinGW working. MinGW, for those not in the know, is a Win32-target GCC with an entire runtime and w32api capable of generating native Windows binaries. It's cool shit. You can build a cross-compiler from Linux that targets Windows.

So, after about six hours of fussing, I got Irrlicht, OpenAL, and libvorbisfile DLLs built and compiled in my cross environment. With a bit more work, I finished #ifdeffing OpenSmash, mostly in the joystick code, and finished the safety checks for compiling on Win32. It built, but didn't link.

Link errors came from Irrlicht. Apparently, between 1.3.1 and 1.4b, Irrlicht changed the API for a certain inherited class, from

virtual bool irr::IEventReceiver::OnEvent(irr::SEvent event);

to

virtual bool irr::IEventReceiver::OnEvent(const irr::SEvent& event);

Which was obviously a problem. To solve this, the source code now follows the latter, and Irrlicht 1.4b (or their latest Subversion) is required to build on both Linux and Win32.

I also started on the DirectInput part of the joysticks. Right now, DirectInput8 is initialized and shut down at the correct time. However, it doesn't do anything more. Also, it doesn't run right in Wine. It doesn't exit when I tell it to; it just spins for a bit and then dies with an error. I'm putting it down to the mangled DLLs I'm using, and hopefully it runs better on a real Win32 system.

Summary:

• Include guards for <dinput.h>, <windows.h>, <linux.h>
• Updated required Irrlicht version to 1.4b
• Added initial Windows 32-bit support
• Added "mingw32" build target to Kdevelop project

See you later. Tomorrow, after class, I tackle the second part of OpenAniDB: Fleshing out the GUI skeletons.
~ C.

OpenAniDB Lives Again!

So, as you may or may not know, I once wrote a tool called OpenAniDB as an interface to the excellent AniDB, a massive anime database. My tool was designed to take care of what I saw as three big problems with the main offline tool, AOM.

1. AOM takes a long time to start up, and takes up a ridiculous amount of RAM. This was not something I appreciated.
2. AOM was Windows-only, written in Delphi. I'm on Linux, and while Wine can run AOM, it's even slower than it is on Windows. Not ideal.
3. AOM's source isn't opened, so I can't fix the problems that I experience with it. Worse, its maintainers need to keep it closed in order to prevent server abuse and are very slow coders, so any bugs stay unresolved for long periods of time.

My answer was OpenAniDB. Written in Python, it tackled the second two problems just by existing. Problem #1 was solved by writing file-hashing code that capped itself at a 10MB memory limit instead of the 100+MB used by AOM.

But, I stopped writing it a while back, due to lack of demand. I didn't want to write and maintain a tool that nobody used. However, the project lives again. I've already revived the code base in C++, and this time around it will be much less hackish and overall more structured.

Since the Subversion at AniDB is low-volume, I will be fully documenting each commit I do, since each one should be jam-packed with bug-squishing goodness.

~ C.

Hello, World!

Hello. My name is Corbin Simpson. I am a programmer and musician.

In computer science, we use a "Hello, world!" program to demonstrate the basic syntax of the programming language we are working in. This post is to let you know what's going to be published here.

I am a programmer. I write in C/C++, Python, and some other stuff, too. I will post updates to projects I work on here, including commentary on individual Subversion commits and plans for new code. Projects include OpenAniDB and OpenSmash. Perhaps more, too.

I am a musician. I will post my music here for you to consume and critique. I'm a jazz pianist with a strong fusion/metal flavor and a love of video games. You might not enjoy what I play, but if you do, I encourage you to share it.

I won't whine about my life here. I have no girlfriend, I don't go out drinking, I have no social life. That's not what this blog is for. This blog is a place for me to put my creative works out for public consumption.

Let's have a fun adventure.
~ Corbin Simpson