Topic: OpenSource tools for OpenSource Reverese Engineering

This post isn't directly related to Fernvale, but is largely inspired by it, so I'd like to take freedom to post about it here, because I'd really like to see it help progress the Fernvale project. I hope moderators won't treat it as a complete offtopic.

So, over last 3 months, I faced 3 interesting reverse-engineering projects - so interesting, that I'd love to work on 2 of them, even despite myself swearing many years ago that I no longer waste my timeon REing proprietary crap. One of these projects is of course Fernvale. Xobs and Bunnie did really great work on the hardest part - to bootstrap open environment on otherwise almost completely closed chip.

Unfortunately, a lot more has to be done to make project more useful than a proverbial Arduino. That work almost likely will include reverse-engineering. And there was a note from Xobs that his IDA database for disassembling BootROM died. Even if it didn't die, it probably wouldn't be useful to anyone else, because demo version of IDA wouldn't allow working with needed architecture, and even if needed arch was supported, demo IDA wouldn't allow to open database. Even if it allowed that wouldn't allow to collaborate on RE in any sensible way, and then in some time database would get corrupted again.

The vicious cycle above goes on for years and decades - open-source reverse-engineering community doesn't learn, trying either to use "industry standard" proprietary tools which offer subpar reusability, or waste time develop own adhoc tools. In either case, results of such efforts are usually very small and fragmented, and eventually go down the toilet.

I also tried to do my final community homework, for example Fernvale presentation mentions Radare2, which indeed getting kind of PR last couple of years. But no, I don't waste my time typing even 1-char commands (followed by Enter), especially if many of them actually required 8+ digit address. And there web interface is hilarious in its awkwardness.

There's no need for all that original research in UI of reverse engineering, there's need to have to have something what Sourcerer offered 25 years go, just open-source, easily hackable.


With the above ideas, I put aside hacking on Fernvale, and instead decided pay off technical debt accumulated for decades, by writing interactive, incremental, direct-manipulation, hackable, high-level disassembler: https://github.com/pfalcon/ScratchABit

The project is work-in-progress. Ironically, I post this on forum related to ARM arch hacking, but there's no ARM support yet. But there's something to look at already (using x86 arch, which is hopefully common ground for everyone). And the whole idea is that it is easy to add *any* arch support, for anyone with even average skills in programming and RE (I obviously won't be able to implement everything myself).

So, I would like to invite every interested passer-by of this forum to give it a look and a try, and hopefully added ScratchABit to arsenal in their tools, and ultimately contribute to its development. Especially I of course would appreciate Xobs and Bunnie considering using it for their next, or maybe even current, hacking projects.

Thanks!

Re: OpenSource tools for OpenSource Reverese Engineering

I'll take a look at it.

radare2 did seem very fragile, and had a really weird UI.  So does everything else, though.

I generally work with ARM, and if it's easy to add platforms I'll give it a go.  I've also been meaning to learn Python.  Having online tools is also pretty cool, and being able to speak gdbserver is a neat trick.  I don't know if you have any plans for that.

But it would be very nice to get a decent, open-source RE project going.

3 (edited by pfalcon 2015-04-25 20:47:08)

Re: OpenSource tools for OpenSource Reverese Engineering

So does everything else, though.

Well, I hate starting from scratch, when dozens of people tried to do it before me with known results - I understand I unlikely would do much better. I have been doing extensive research, and the closest thing to something usable I found was https://github.com/fpw/kianxali , and I actually contributed few patches there. But otherwise it shares typical issues: written by a guy for his graduation thesis, project future is unclear; written in Java, so compilation overhead, verbose language, etc.

So, I decided to give my go at it after all the years of wishful thinking, trying to explicitly avoid the issues I saw with other projects (but of course adding my own, d'oh).

I generally work with ARM, and if it's easy to add platforms I'll give it a go.

That's one of the thing I decided to do differently - instead of hoping that people would write stuff specifically for my piece (not going to happen), allow to reuse stuff already written. Generally, the way it started is that I took Xtensa CPU plugin (https://github.com/themadinventor/ida-xtensa), and wrote code around it to make it disassemble something for me, then added very bare UI to drive it in direct-manipulation way (press a key, immediately see a result). It should be noted that IDA API to write plugins is very ugly (e.g.: http://blog.delroth.net/2011/11/random- … -module/), but again, lot is written for it (including open-source), so you can find many things, and take something as an example. And even if one needs to learn a bit of that API, one shouldn't see the effort as wasted - after all, the API of leading, industry-standard tool is being learned ;-).

I've also been meaning to learn Python.

I truly believe that's one the most readable, writable, and easy to learn languages around (why I for example work on MicroPython to make it scale down to even modest embedded environments).

Having online tools is also pretty cool,

My idea is to add git integration, so when you save it commits changes, and possibly pushes them (the database is already in text format) of course. Then people submit pull requests to each other, working concurrently, while still doing reviews, maintaining attribution, etc. I don't believe in game-like "crowd" online services - just yesterday I stumbled at such service which was run by some company later bought by Google - kaboom, all stuff you relied on is gone. I also know open-source tool in such vein, and I had hard time arguing to its author that his flaming tool should not try to do network connections without notifying me first. All in all, if there's no explicit control of what *you* get from outside, the only thing such services are good is for spreading poison.


and being able to speak gdbserver is a neat trick.

I know a lot of people use IDA in "live debugging" mode (for malware analysis, etc.), but I never did that, so don't have an itch for that ;-). Otherwise, I have interest to integrate it with emulators, symbolic executors, decompilers - just very little time to do any of that. So, my idea is to implement minimum functionality which would satisfy me, and maintain it in a shape that other people could run it easily enough and see something before it crashes on them, then wait for patches ;-).

Re: OpenSource tools for OpenSource Reverese Engineering

I didn't realize you work with MicroPython.  Neat!  I just did a quick-and-dirty port of it to Fernvale.  Thanks for a cool project.

By "online debugging" I meant more speaking to gdbserver and less gamification and "share this on Facebook".  I only really used it once, and that was when I dumped the ROM of Fernvale, threw it into qemu, added hooks so that reads/writes to register space went to real hardware, and attached IDA to it.  That was really handy for figuring out what registers were at least important.

I do agree though that writing IDA plugins is atrocious.  Even setting up the development environment takes ages, and it's unclear what C compilers actually work.

Looking forward to a fancy tool to add to the toolbox.

Re: OpenSource tools for OpenSource Reverese Engineering

xobs wrote:

I didn't realize you work with MicroPython.  Neat!  I just did a quick-and-dirty port of it to Fernvale.  Thanks for a cool project.

I actually left a comment to your commit that porting micropython was the idea why I bought an mt6260 device and started to look into fernly. Thanks for starting the port - all this time I hoped to try it, but didn't get to it, unfortunately. Let us know if/when you think it will be worth to merge in the mainline, and i surely hope to get to work on it too eventually.

By "online debugging" I meant more speaking to gdbserver and less gamification and "share this on Facebook".  I only really used it once, and that was when I dumped the ROM of Fernvale, threw it into qemu, added hooks so that reads/writes to register space went to real hardware, and attached IDA to it.  That was really handy for figuring out what registers were at least important.

I see, I'll surely keep this feature request in mind.

In the meantime, work on ScratchABit continues, for example I've added an initial implementation of ELF loader and work on elaborating it, while also working on UI functionality.

Re: OpenSource tools for OpenSource Reverese Engineering

Ok there was no update on ScratchABit for a while, but there always has been background, intermittent, but still work on it. Finally around October last year, after 1.5 years of (such intermittent) development, it finally reached featureset I myself find suitable for sustainable use. It took another few months to make a new release, but ScratchABit is at 1.4 now: https://github.com/pfalcon/ScratchABit/ … s/tag/v1.4 , and 1.5 hopefully will follow soon.

I know that few people gave it a try, but it didn't work in the way I'd like it to be: I hack on ScratchABit, and other people start/maintain RE projects using it. So, I gave up and published my own project to reverse engineer binary blobs in ESP8266 firmware: https://github.com/pfalcon/xtensa-subje … -p20160809

7 (edited by pfalcon 2018-01-08 06:35:29)

Re: OpenSource tools for OpenSource Reverese Engineering

ScratchABit is at 1.8 btw: https://github.com/pfalcon/ScratchABit/ … s/tag/v1.8 . Still no solid ARM support, though some ARM Thumb (not Thumb2) plugin was contributed.

https://raw.githubusercontent.com/pfalcon/ScratchABit/master/docs/scratchabit.png

Re: OpenSource tools for OpenSource Reverese Engineering

Still no solid ARM support

Ok, it's time to have that resolved. The initial Capstone plugin is now in the master, and adding only-ARM and only-Thumb disassembly was easy. But for any real-world usage, interworking support is needed, i.e. ARM and Thumb code mixed in the same binary. Supporting that is more challenging, and require some redesigning of the core. I'm working thru it, and would appreciate any testing in the meantime.

Re: OpenSource tools for OpenSource Reverese Engineering

Ok, I guess I finally got it right in the latest master. It's now possible to disassemble binaries containing both ARM and Thumb code. After additional testing, that should become 2.0.