Know your machine

We found out, for example, that the byte code $2C on the C64 was able the cover the next two bytes, means render them useless. The allowed us to create subroutines with three different entry points, like

entry0:		LDY #0 
	    	.byte $2c 
entry1:		LDY #1 
entryY:		… 

To jump the routine at entry0 would start with a zero, at entry1 with a 1 and on entryY we would have to set Y ourselves. The simple trick was this: LDY #num was encoded as two bytes A0 <num>, so LDY #0 would be A0 00, LDY #1 is A0 01 and so on. The code 2C means BIT <address> and is encoded 2c LO HI as three bytes.

If you jump in entry0, the processor would execute LDY #0; BIT $01a0; … means the second Y-load would not be executed but instead taken as argument for $2C, the BIT opcode.

Bit does nothing more than an AND between the byte at that address and the ACCU, without putting the result anywhere, only manipulating the ZERO flag, means you can check after that if the result would have been zero or not.

This sequence just needed five bytes and encoded three possible setups for a function and is a good example for the principle that an engineer should know his machine.

We did know our machines to the last byte of it’s “operating system” at those times. I still have that thick folder of commented core dump of C64 ROM lying around where I scribbled additional notes into them. We were analysing that and tried to find out how we could reuse some of the code to shorten our typical algorithms with them.

And we did. A friend of mine did one day this neat little trick to jump into one of the BASIC routine to set up and calculate some formula, but instead of letting it run to the end he had programmed a timer that was interrupting the routine and jumping back into his program. For that he was counting the cycles that the program would need for that.

This was all assembly-level programming, totally anarchic and we didn’t respect entry points, exit points, stack values and to write self-modifying code was not considered “bad” but instead “elegant”. And still, if I would like to write code that was impossible to reverse engineer, I’d use those techniques.

What is self-modifying code other than optimized JIT compiling? It’s the same. Just that you don’t JIT the whole thing but only the critical parts. And JIT is now valued oh so high, but self-modification is not? Why? Ask that yourself, or you might find an answer in the ongoing article.

New Languages

When I read, that some people consider game programming with Lisp code in it “efficient”, I really get red points in my field of view. Instead of following some crappy languages from the 60’s we built our own languages on the fly. It was normal in those days to invent your own little language for a problem, write your own interpreter or compiler or most often we just did little virtual machines. Not, of course, if we wanted to be fast, but if we wanted to be short. 30p21+ is very short for result=30+21 and especially if you don’t use crappy ASCII but BCD data for that: four bytes.

This “inventing languages” was really a normal act back in the days and it was a very very efficient way to do things in very small space. As long as the problem wasn’t possibly done shorter directly in Assembly. Which in the case of my example, sadly, would nearly be the case: LDA #30; CLC; ADC #21. Sadly we have to clear the carry, and that’s five bytes. So… But you always had to add to the size of your problem the size of the virtual machine and if you really are shorter by that.

The more complex the problems got, the more efficient this way became. Well, not the fastest way, we are talking about size.

Data packing

Instead of ASCII numbers we were taking BCD numbers, if we only had texts, we used five bit, not eight and encoded them in a 5-bit bitstream. Inform did that, the Infocom adventure language, as an example. Later we did, naturally huffman encoding and combined it with run-length encoding to get smaller sizes. The typical fast-loader routines on the C64 worked with that.

Also self-depacking code was the normal case, means we had a compressed part of code, loaded it and de-crunched them. Sometimes it was self-modifying code that de-packed itself, sometimes we used loader routines. You see that very often on the C64.

But more efficient than huffman or other typical packing algorithms (that had to be small!) was a cellular automata. What we often also call “procedural patterns”. This was used not only in graphics, but also in representing the whole game environment. Like “Elite” of David Braben did that. The whole universe was just a little formula that extracted thousands and thousands of unique planets from a little loop of code, that did run with a kind of specialized pseudorandom generator.

The world would never have fitted into memory and many people think, that cellular automata is the next step of packing. If we find a world, that is created from one of those automata that is exactly like the destination data we want, we have the most perfect packing algorithm. Indeed, this problem is not solved, so people tend to use it the other way round: “Let’s see what our generator produces and see if we can make that a world.”

In effect we used every bit we had. And that means, if I come back to my BCD calculator, that if we say that bytes $00 to $99 are the numbers 00 to 99 There are the numbers $AA to $FF that are unused and that we would start using for commands for our VM. To break it down to a single digit 0…9 are interpreted as a number, A…F are the six commands our machine knows. push add sub mul div ext, while ext would extend our command set for whatever we like.

30 A2 1B

That’s three bytes for the addition I described above: 30 push 21 add

It’s hard to do it any shorter, especially if you run something like a cruncher of that after all is done.

Reusage

We did reuse code and dataspace all the time. So, somewhere we had a little codespace and we de-crunched the next routine into it after the first ran out. Usually the whole process with what we did that was our own special little language that we invented exactly for this purpose. If you do that like this, you have 99% of your code in a packed form in memory, while only the de-packer and VM has to be executable.

Self-modifying, as I also already mentioned is not “ugly” or “bad”, it’s indeed beautiful and elegant if you have to solve those problems. There is no bad tool in the toolbox of an engineer, just the wrong tool for a specific task and the right tool.

And those tools were the right tools for this. I would always do this again, if I had to solve problems like that. The BASIC interpreter on the C64 was done with self-modifying code.

That’s very sensible, efficient and fast code; this is the CHRGET routine of the BASIC interpreter of the C64. Like all interpreters of Commodore it was put into the zeropage, because the access was faster there. You could jump into $73 to get the next BASIC token, you could jump into $79 to get the last token and at $7a and $7b you see, as the argument of LDA the position of the next token.

This little routine was copied from ROM to RAM during setup. And it was absolutely possible to mess around with it and do things to your BASIC that the engineers never expected to happen.

We of course did that. And we learned from that.

The dispute about if that is a good or a bad coding convention aside, there is in this case nothing wrong with that. It’s clearly to see what they are doing, it’s efficient, it’s clean code, it’s very sensible.

They were engineers, they had a task to do something impossible: to bring a kind of operating system to computers that originally were designed for control circuits. And they had to make a language work, that at that time in competition to huge and mighty mainframe languages like Fortran or Cobol. They had to be faster than they were, tighter, better, and more user-friendly.

And all in all they did that. They killed their competition. And for a long time after that, engineers were writing their routines in BASIC on those machines, not on mainframes, not on the languages of the old men.

This young punk sent all those into retirement. I know that well, because I was a hired coder at that time and I did at the age of 15 to 20 a lot of coding work for German engineers in BASIC.

Where has this “structured-programming” and the “bads” and “goods” of coding practice got those old men? They got them into retirement. And we young punks were taking over. On machines that were like factor 10000 cheaper than their machines, faster, leaner, cheaper, smarter, better. And easier to use.

Revolution

It was a revolution a revolution where we still can sniff the smoke in the air from all the aged things we did burn down at that time. When kids, really literal kids were doing man’s jobs, where you didn’t have a chance to get your feet into the field if you were too old, because you were not in the scene, you didn’t get the support of your whole generation. With 25 at those days, you were fossil, well some said with 20 it was hopeless, but I wasn’t that radical.

The engineers we were working for were young and eager and they took the power from the high castles of dinosaur computers and brought the power back to the people, back to the little engineer in his bureau, with the help of a hand full of us kids.

BASIC and the 8-bit era, that wasn’t a revolution in gaming or something. Not as I see it: it was a revolution in engineering. My personal mentor, who was a construction engineer at that time started to take over the work from all other engineers in Hamburg with my work. I don’t know how many of the big firms did bite the dust with him and my code. But they were a lot.

And the half a million worth mainframes were sold as scrap metal before the 80’s did run out. We totally annihilated them. We took them by surprise, we destroyed a whole infrastructure. An infrastructure that was towered behind incredible walls of certifications and years of education, of machines that never cost less than in the 10k, more in the 100k or 300k range. We took them with $200 machines. We took their expensive and walled-off knowledge and put it on our internet: for free. That’s revolution!

Like the Russians did with the invention of the Blade server: just use cheap little PC hardware, because the all-mighty US empire won’t deliver their mainframes. Funny thing: even if the blade server was invented by Russian hackers because of the technology blockade during cold war and after that, a US firm now claims to have the patent on them. Which is hilarious. And just feeds my agenda to kill all patents; they are slowing everything down, don’t help engineers, tower the mighty and only trolls have them. Only unproductive vermin use them to extort money from the productive part of the society. Let’s kill that shit.

The Chinese did that right: they didn’t accept the patent system. And we shouldn’t do that, too. It’s overcome, it’s 19th century, it only feeds the vermin.

With this export blockade, they destroyed themselves, because they cornered the hungry and eager little rats and because of need of survival, they did this innovation and with this innovation, they killed them. We killed them. It’s the way I’m still working today: a hungry little cheapskate that has to work with less and has to work more.

That has always be awake and eager to learn and hack himself into the difficult corners of technology and find new ways, where the fat guys just load bloatware like Bootstrap, just to drop one of the most terrible ones that should be burned and eaten alive and every company that is using it should be overrun and eaten by us rats.

And I am proud of the destruction we left behind. Because we did all the things they said we must not do, because they are ugly and unclean. I hate the word “unclean”, we always try to declare a winner of evolution as “unclean”, to save the old overcome things as something better, something of a higher moral value. To arrogant down from towering moral high ground. Like we call the rat or the pig “unclean” animals, even if those animals are exceptional social and if they live in a healthy environment they are some of the most clean animals there are. And they are smart and eager and hungry. They are the winners of evolution and they were feeding from the carcasses of the dinosaurs of their times like we, the generation of cyber-rats were feeding on the machines of the old fat men. We really killed them. And we still do. Because we work with less. With less hardware, with less energy, with less money and with less fucking certification.

That’s all just methods of the big to keep the small and hungry out. Competence comes with doing something, not with certification on the walls, not with standing on the podium and preaching some ideology; the geek will inherit the earth.

But that’s not how engineering works; you get shit done. And science also only accepts working concepts, evolution does only accept working and efficient concepts. If you try to tower yourself behind ideology and paradigms you are soon a goner.

Evolution will sort you out.

Change

We are living in a changing world. And the techniques we learned and done back then are coming back, when the next revolution comes. And it comes with the new IoT, with the mesh networks, the little computers that pop up everywhere. Not for $1000 but for $20. That’s the revolutionary step we had back then. That’s the revolutionary step that is coming today.

And with that the methods we used to overcome the limits of our hardware; smart algorithms, data packing, procedural creation of content, cellular automata, dynamic code, means self-modifying as well as reusing code-space, data-space and mixing up both. Putting code into dataspace and vice versa. Knowing our machines to the metal, to the last bit, working critical routines in Assembly, using our GPU and other specialized chipsets. And doing funny things with combining those principles to something new.

That’s the meaning of hacking: doing incredible things by finding new ways to do something, finding new aspects of principles or tools we already have and how to combine them. Finding points where the old men are producing fat and bloated crap, out of laziness or because corporate code is fat code, because corporates are fat and lazy. It’s the rat that will win the evolution. We are the rats.

David vs Goliath

We are cheap, disposable, we can fill even the smallest niche, we are fast, quick, cunning, we don’t shy away from doing “dirty” tricks, which is just an arrogant way to describe someone doing a thing better than they are doing it. “Dirty” never describes quality. It always is the description of the arrogant and mighty about the new things that are coming, about the young and hungry. It is always how Goliath will see David: Davids never play by the rules.

Because the rules are made by the mighty, for the mighty.

And to close my laudatio to the new engineering and to get rid of the nay-sayers, here something that I found very readable:

How David Beats Goliath

Please read and enjoy that article. It’s about us, the rats. In the form of little blond girls, playing basketball against towering semi-pros. Same thing.

Dirty little code rats

And I feel a lot of pride, when I call myself a “dirty code rat”. It means, I’m new, I’m lean, mean, hungry, I don’t play by the rules of the Goliath-system and I’m winning the competition. And I’m feeding from the decaying flesh of the fallen, that we brought down. Let’s all be “dirty” little rats, they fear us. It means really power to the simple man, to the street, to the underdog and take it away from the big ones. Because they can’t shine where we can: play outside their rules.

And I don’t mean “outside the law”, because that’s lazy. But as engineers that means to deliver the goods the “clean” tools can’t deliver, can’t even dream of. Try to write something serious in lazy Haskell or Python or Lisp or Scala or Java or whatever; won’t happen. That’s all tools for the giants that have enough of everything. We don’t. Let’s destroy them, let’s eat them. Let’s take over. Just by being better. We are dirty little code rats, we can even live in the walls of their castles. We can grow in numbers even from the bread crumbs that fall from their rich tables. And then we bring them down.

Like we did in the 80’s with mainframes, like we did in the 90’s with super computers. Like we did in the new millennium with our gaming GPU and our $300 Playstation computers that’s worth more than double the hardware costs. Like we do now by killing the fat of HLL by coming up from the bottom with C and ASM, that will gnaw and claw on their fat and lazy code base.