The following is a selection of blog posts from the development of PlayBasic to DLL (Machine code translator) and the parallel upgrade PlayBasic V1.64P
After much head scratching it was decided that the VM's instruction set needed a few alterations with it's looping instructions. These changes are two fold, the parser now does a better job at producing cleaner input parameter data and the instructions produced are now more stream lined. Producing better code, is better for everything down the line from running the code through the VM and thus any translations. The translation stuff is really why these changes need to be made now, since the original parser would use generic instructions where some data was pre-computed. From the translators perspective, this made it very difficult to accurately detect those loops in a single pass.
Today's changes introduce some new high level instructions to the VM. These changes make it trivial to accurately pin point the set up code in a FOR/NEXT looping blocks. I've already added a decoder to PB2DLL which locates the new opcodes and builds a export string in the dissassembly. The detection and output are working fine, but it's here that I noticed there's a few extra moves creating into the output code than need be. Which mostly happens when a loop is using literals. In previous editions the parser would drop an extra move prior to the loop entry point. The same can happen in the destination value expressions also. These things are easily tweaked out, just hadn't noticed them before.
PlayBasic V1.64P Beta 38 - For / Next Loop updates (April 22, 2014)
If you didn't already know, it's been Easter holidays here which has just come to a close, so only started firing up the compilers today. Last time I was working on the tuning the compiler and VM to use the new looping setup codes for FOR/NEXT loops. These changes have were translated across to PB2DLL when I was happy with the new structures. The changes are much easier for PB2LL parser to accurately detect what's going on in the program in regards to loops. But not only that, the new format allows us to make more optimizations to the assembly.
Tonight I've been back working on the parser in the PlayBasic compiler, as when converting the code to assembly, there's often these extra move operations that would end up in the byte code first and thus translated assembly. Surprisingly it generally happened what a for loop's (start / end) were both literal. For some reason it'd drop them into temps and then pull them into the loop structures internal registers. Which appears to be some left over instruction set behaviors that are no longer necessary. Older versions of the VM worked differently than today, in particular when dealing with literals. Now they can be treated just like variables, so don't they need to be treated special ways.
Something you might not be aware of is that when a for/loop is started, there's a comparison between the Start and End of loop. If Start is bigger than the end, the execution jumps to the beyond the matching next.
For example,
PlayBasic Code:
For lp =0to100print lp
next
So there's an indivisible screening wrapped around loop entry, which might look at like this in PB style code.
PlayBasic Code:
if0<=100For lp =0to100print lp
nextendif
One of the new benefits of the FOR/NEXT replacement opcodes, is the export parser can detect constant behaviors and produce code accordingly. So the comparison behind the Start /End loop counters can actually be made at compile time when they're both literal. Removing those lead in comparisons from the output code. The same goes for the storing them in internal VM registers, saving more set up code.
Being able to trap literals helps with STEP loops also. When a variable is used in for/next/step loop, the looping structure has constantly query what sign the STEP, then clip the loop counter based upon it. If the sign of step is known at compile time, then the loop can be created with this knowledge ahead of time. Removing more redundancy from the final translated assembly (in PB2DLL)
PB is pretty flexible with FOR/NEXTS allowing integer and floating point counters with variable step directions in both. People rarely use floating point loops, but since the VM supports it, the exported assembly through PB2DLL should also. Dropped in the standard by one loop structure earlier today and have just started adding the variable STEP stuff with floats. It's painful purely because of the horrible fpu instruction set in x86 chips...
PlayBasic To Dll - Tweaking Looping Constructs ( April 24, 2014)
The past day or so I've been wrestling with FOR/NEXT/STEP implementation in the translator of all things. Those reading the PB1.64P upgrade blog lately would no doubt be aware that the compiler/VM side has recently changed, the changes were explicitly made to facilitate an easier more accurate translation to machine code from the byte code origin. When porting the changes across to PB2DLL I noticed that it only supported Integer FOR/NEXT looping structures, where as the VM supports integer and floating point look structures
Floating point loops present a bit of a problem due to the FPU's rather awful implementation, but really that's everything floating point in these things. Writing the translator code isn't difficult, it's just that it needs extra steps when dealing with floats which feels rather awkward. Knowing this, invested some time into the PB compiler to make sure the compiler side cleans out any unnecessary steps when setting up the loop and gives enough information to easily work out how to set the loop and what kind of set up is needed. The decoder is able to pick up short cut situations where you used literals in your For Next Statement and cut down amount of screen code needed, in some cases on screening is need. Since the loops duration can be validated at compile time. It's those little things that help fine tune the performance of the final machine code.
The big question is when do we release ? - Well since both sides need to be in alignment for the resulting DLL's work, the original plan was to release them both together, but I think we might release PB2DLL first.. This will help further test the translation in real world situations. Of course that'll most likely mean that there will be a few updates made prior to the release, but there's not that can be done about it. It now just needs people other than me testing it.
PlayBasic To Dll - Exporting LinkDLL binding code ( April 28, 2014)
Had a few short programming (testing mainly) sessions over the weekend. Up until now, I've been testing the translator from the PB IDE. So in order to run it, i've got to compile and run as normal (F5), but as release time nears, we need to be able to build it into a final EXE, but it's here we ran into a new problem, where the exported internal structures could stop the exe from running. I though this might happen, given how much of the PB VM internals have changed over the last 6 months. Tracking down the issue took a while, but was fairly easy to solve which is a good thing. So now I can build & run PB2DLL in exe form and it works as normal.
Once the build problem was sorted, i've been getting back to the GUI and internal exporter functions. For the first version(s) I'm not going to bother with adding switching over the assembly code generation/optimization, it'll just do it silently. There are times when that wouldn't be wanted, but can't foresee anybody running into such problems. Which means for the time being some of the GUI toggles have been ripped. The only option that is useful is the Export LINKDLL code option. This option gets the translator to build a chunk of the PlayBasic source code that handles the function binding to the newly created DLL.
Linking to DLL's is something of a mystery for most people, basically all that happens is the PB compiler/VM create interfaces to external machine code that look and act like normal PB functions in your program. There's some limitations in current versions of PB as to what data type/structures you can pass into and out of them, but those will be tweaked out in time. In order to use our newly converted DLL's in our PlayBasic programs, it stands to reason that we'd need to know how to write the LinkDLL binding code for the functions we wish to use. Which was true yesterday, but now the translator can build a template of the exported functions for you in PlayBasic code. Obviously there's some limits in terms of data types that it currently supports, which stem from the PB compiler side not supporting passing PB internal structures through to external code. Why? - because nothing other than PB understands PB internal structures, making supporting that originally point less.
To test the LinkDLL exporting, added a few bogus functions to the common point rotation demo and here's what we get.
This code can either get cut'n'pasted in your project or #included. Either way once it's in and assuming PB can find your DLL (it's in the project folder) then your away in the machine code era!
What might be a good is to get PB2DLL to mangle the export names in the DLL. So if you have a function called "AddIntegers", the external name can be mangled into some random hash, such as 'ABHDGSNFSSF' - This would mean that exported linkDLL codes must match the build of dll exactly, as if the alias names don't match, windows can't local the machine code function and it won't run ! It'd only be helpful to prevent snoops from look at your function names and potentially working out the parameters.
Beta #39 introduces a bunch of subtle but equally important changes, starting with the newly replaced FOR/NEXT/STEP parser and VM instruction set, there's also a few changes to the VM close down behavior. The close down changes are due to a strange issue i've been having with PB2DLL where the PB2DLL EXE would crash on exit when the program wasn't registered (worked fine once you entered your serial).
So I've been chasing the close down issue most of the day, it appears that some typed arrays can somehow contained data that wasn't a typed array container. How that's the possible is a mystery at the moment, but I've included some more screening in the clean up code to make sure it's releasing the array an not some of random bank of that data that something else was/is using. Getting this right is more important now than ever, thus is because of the way type deletion works in the runtime, so we have to make sure it's releasing data that makes sense, as if it doesn't it'll die.
The real question is how is the bad array data was/is getting into the arrays in the first place, which i'm not 100% on at this point. So i've added some debug messages to the debugger console (F7 to run, then click on the console tab) to help the community track where else these quirks put up.. Does it happen at runtime ? or only closing down the VM..
The past day or so I've been wrestling with FOR/NEXT/STEP implementation in the translator of all things. Those reading the PB1.64P upgrade blog lately would no doubt be aware that the compiler/VM side has recently changed, the changes were explicitly made to facilitate an easier more accurate translation to machine code from the byte code origin. When porting the changes across to PB2DLL I noticed that it only supported Integer FOR/NEXT looping structures, where as the VM supports integer and floating point look structures
Floating point loops present a bit of a problem due to the FPU's rather awful implementation, but really that's everything floating point in these things. Writing the translator code isn't difficult, it's just that it needs extra steps when dealing with floats which feels rather awkward. Knowing this, invested some time into the PB compiler to make sure the compiler side cleans out any unnecessary steps when setting up the loop and gives enough information to easily work out how to set the loop and what kind of set up is needed. The decoder is able to pick up short cut situations where you used literals in your For Next Statement and cut down amount of screen code needed, in some cases on screening is need. Since the loops duration can be validated at compile time. It's those little things that help fine tune the performance of the final machine code.
The big question is when do we release ? - Well since both sides need to be in alignment for the resulting DLL's work, the original plan was to release them both together, but I think we might release PB2DLL first.. This will help further test the translation in real world situations. Of course that'll most likely mean that there will be a few updates made prior to the release, but there's not that can be done about it. It now just needs people other than me testing it.
The following is from a selection of blog posts from the development of PlayBasic to DLL (Machine code translator) over the past months.
PlayBasic To Dll - Joining The Dots ( April 08, 2014)
The GUI side of the app is looking fairly complete now (apart from some tweaks), wrote (largely cut'n'pasted) a history dialog the other day from one of the other apps. Turned it up a bit, one of my pet peeves is programs that only remember the last couple of files you worked on. The current solution has no max size, but really it's unlikely you'll ever have more than a page or so loaded. There's a clean up button to remove files that no longer exist on disk. So when stuff get moved or delete or whatever.
The other remaining GUI tidbit that was left over, was the batch / folder scanning stuff, which was just transplanted across from the Amos2BASIC converter. The batch processing stuff just scans the folder you provide looking for PBA files. A PBA file for those who don't know, is the PlayBasic source code file. Each source it finds it'll attempt to build into a DLL. The translator expects the sources to be stand alone at this point, so they currently can't have #INCLUDE statements in them. Which is limitation, but It should be able to be overcome.
In order to get the best out of the tool, it's highly advisable programmers adopt a parent->child structure to their machine code accelerated projects. So your main project folder contains the game / tool source + media as normal, in here you'd also have any secondary library sources stored inside that main folder. Your libraries folder would include your PlayBasic DLL projects sources. So if you have a couple of lib's you can use the build folder option in PB2DLL to run through and build everything for you in one go. The build process will write the created DLL's to those folders and ideally generate linkDLL wrapper code for them. So in your main source you just #Include those wrappers and your away. .
So if we had an imagery a game called "PONG" , we'd make a project folder called PONG and hopefully give it a meaningful project name such as PONG.PBP (PlayBasic Project file), rather than the default 'Project.pbp' which the IDE will assign those lazy people. Inside that folder we'd create a MEDIA folder for all the games graphics and sound assets. This simple structure immediately helps keep everything more organized from just file management through our actual source code. If for some reason we wanted to add build a DLL library to help our pong game, we're create another folder inside the PONG folder. It could be called anything, but giving it a name such a DLL's or Libs, or Libraries or something, something that should be obvious to what the folder contains
Now let's say in your pong game you're loading media files that for of a different format, you can load them in your main source code, but there's some VM overhead in doing that. This would be an ideal situation for converting the loader/conversion code to a DLL. To do so, you start a new project in PB, then we save this project (with a meaningful name) into our games / dlls folder. So if our library was to be a packing library we might call it "Packer", where it's path might be PONG/DLL/PACKER/. Once we have a project, we'd cut'n'paste all the required code for the DLL functions to operate separately. So any structures arrays etc your packer functions need, need to be inside that project. The project should compile without any external requirements. Once you have that, directly PB2DLL to that folder/and source and translate it.
Once the functions we want are externally built into machine code, we need to convert our parent project to use the DLL version of the functions, rather than the original versions that might be in our code. In this case, we'd pull all the Packer functions of our game. We don't need them in the main source anymore since all that code is now hidden away in our machine code dll. However, if you're passing types between the sides you would still need those Types to be declared within your main source. PB can't get this information back from the DLL. Moreover the DLL is like a black box, so the only functions you can see inside the dll are those that you exposed in your code. To expose a function we rename it with the "DLL_" prefix. Only these will be exported. Any function not containing the prefix, will be included inside the final code but not visible from the outside, so we can't access it.
PlayBasic To Dll - The last 10% is really more like 90% (April 09, 2014)
One of the many frustrating things about programming is the almost there sensation, where the last 10 or so percent of the project, feels like it takes more like 90% of the development time. Testing is notoriously difficult and not to mention time consuming, one of the challenges with a project like this, is the near infinite amount of input situations the program has to deal with. One little issue I've been having is getting PB2DLL to control and understand the responses from both the PB compiler and FASM. It's fairly easy trapping a compiler errors (Shown bellow ) from the PB and returning this information back to the user, but from assembler is still a bit iffy trying to sense of what it returns, which soon goes beyond the scope of the product. Obviously it makes more sense to trap errors during generation, rather than after assembly. Since it's entirely possible a broken translation source could assemble.
Another problem that's been appearing is with batched conversions, where on some sets of sources, it's happy to compile and convert them, but there's the odd program that doesn't like being compiled in a batch, they'll compile fine alone, but not in a group. So there's something wrong with the loader state not being refreshed, but just can't find the actual smoking gun.
PlayBasic To Dll - Eureka, It's Finally Working (April 11, 2014)
There's a funny thing that happens when your chasing down bugs, where you'll follow a hunch down a rabbit hole almost infinitely, regardless of if there's real basis in it or not. You know it seems like this, so it must be this! Now the past week or so, I've tracking strange faults in the translator where compiling the same program twice in a row would work, then it'd fail when changing sources in a batch. Initially it seem GUI related, since the test code worked fine in the stand alone translator, but generally failed when you batched a folder out (translated a group of sources). There's a few obvious things come to mind, like the translators not initializing it's internal structures properly, or there's some unforeseen collision somewhere.
Initially focused on tuning up the initialization code, but the problem persisted.. So off I go looking for any collisions or potential issues between the two slabs of code, such as clash of variable scopes. Even found a couple of such issues, which looked potentially lethal, but turned out to be a bust.. Next logically conclusion, oh there's must some bug in PB that's at fault.. But right before diving into the runtimes, it seemed it might be nice to get a clearer picture of what the translator is looking at to begin with. If that code isn't valid, or has some bogus data in it, then there's possible smoking gun and wouldn't ya know it when comparing the byte code state being compiles, there's an odd similarity in where the errors occur in them, in particular in the map that flags byte code as data and to be skipped.
So off I go looking at the byte code loader function and hey presto, the function REDIM's the ignore table. This creates a problem as Redim preserves the contents of that array, the array is used in parallel to the main byte code to screen stuff out, so compiling the same program would work but if you compile program A then program B.. Program B has Programs A's ignore list applied to it, the more programs to built in sequences the of lists get merged creating all sorts of strange artifacts. But right on 5PM that's bee been fixed now and it's working nicely !
Haven't just been searching for bugs in the translator, I've also added more error trapping abilities. Earlier this week, added support to trap error codes form PlayBasic compile process. This returned a line number / error number and error message, but it couldn't get the source line, until now. So when a compile error occurs, it'll dump that to the console also. Which should give you some idea as that's wrong with the source. Beyond that, there was no error trapping in the translator side of the tool as all. In fact if an error occurred, it'd still call the assembler and try build your DLL. PB2DLL has limit on the amount of errors it'll accept, unlike PB. This is because it's possible the error message you're getting is just a warning, so the code may assemble, but it may not also.. If there's too many errors though, it just considers this fatal case and aborts the process completely.
Anyway, prior to this week I was hoping to have the first commercial releases up and running by the this weekend, the delays this week make that very doubtful now, but it is just around the corner !
PlayBasic To Dll - Testing - 1,000,000 Point Rotation Example (real time) (April 11, 2014)
Here's something for those die hard skeptics out there, is rotating / clipping and drawing a 1,000,000 points to the screen possible in PlayBasic ? - Well, it's not running through the VM's, but it is if you translate your code to machine code via PB2DLL then it's easily within your grasp.
I remember back in the mid 90's counting clock cycles to try and get a few hundred, let alone a 1000 points rotated in a couple of frames from hand optimized assembly. You can do that easily in PB today (even running on the slowest VM), but it's mind boggling how far this has all scaled up.
The best news about this, is the current PB2DLL's translator has only a few redundancy tweaks in it, as yet I haven't added any arithmetic optimizations.
The following is from a selection of blog posts from the development of PlayBasic to DLL (Machine code translator) over the past months.
PlayBasic To Dll - User Define Types - Addressing the same ground. (December 15, 2013)
I've been testing the translation tonight hunting down those little gotchas in some bare bones type demos. Found one such issue where PB would crash on exit when a typed array created inside the dll was returned to the VM. Initially was thinking it was caused due to some difference between the Undim process in the VM, which is called when the VM exists. This was a close guess, but it turned out to be just the order in which some of the vm libraries release themselves. VM would release any bound resources such as DLL's before it release the array memory. This is no problem when all the array data is created locally in the VM, but it's a big issue when the arrays point to structure definitions inside the DLL. Which would cause access violations and hence the crash.
Today's build gets us to a point where the DLL side and VM side can create a type and other side can release it. This allows the flow of data from VM to DLL and DLL to VM more freely. Even so, i'd still recommend at least trying to keep the data life cycle management to one side or the other, but it's workable if you can't wrap your head around that. If this all sounds familiar it's because a few months back I had to rewrite how the VM deals with it's local types. In the older editions the VM maintains one central list. Which is fine in a world where 3rd party code isn't expected to plug into the VM, but bound DLL's need to be able to control data much the same way the vm does. There's other solutions of course, but altering it to support host structures seems the most workable.
All this messing around means you can finally do this, bellow we have a function that we'd export to Dll and bellow that is some code that accepts the returned structure..
PlayBasic Code:
// Declare structure on DLL side.. // The VM will need a copy of this also if you// indent to look at the fields in the types.. Type Dude2
xxxx,yyyy,zzzz
string$
endtypeFunction DLL_FillArray(Size)Dim Cool(size)as Dude2
For lp =0to size
Cool(lp)=new dude2
Cool(lp).xxxx=100+lp
Cool(lp).yyyy=200+lp
Cool(lp).zzzz=300+lp
Cool(lp).string$="This String "+Str$(lp)nextEndFunction Cool()as Dude2
The dll code just creates a junk array and fills it with some data, before returning the array handle upon completion back to the caller, the VM side.
PlayBasic Code:
linkDLL"testdll.dll"
DLL_FillArray(Size)alias"fillarray"asintegerEndLinkDLL// Declare structure on VM sideType Dude2
x,y,z
Name$
endtype// Dim our reciever array as type DUDe2Dim K(0)as dude2
// Call our DLL function and assign the array handle to our K() array
k()= DLL_FillArray(10)// query the contents of itifGetArrayStatus(k())for lp=0toGetArrayElements(k())print k(lp)print k(lp).x
print k(lp).y
print k(lp).z
print k(lp).name$
nextendifprint"done"Syncwaitnokeywaitkey
The VM side is just running through the contents of the array and dumping them to the screen. Not much of an example, but it's fairly seamless. There's a couple of oddity that appear in the debugger if you looked at the contents of the k() array in the VM after it was created externally, but those seem from the DLL's dimension instruction not classifying the arrays structure. So it doesn't know the original 'name' of the type.
Edit #1 - Debugger Name Resolving
One problem i'd been having with structures passed back from a DLL was that if you clicked on them in the debugger the structure names would be wrong, but that seems to be now fixed.. As if you got to variables and click on k() array, it drops the info into the panel as normal.
PlayBasic To Dll - Redim - Life cycle of arrays & LIsts ( December 18, 2013)
Pretty much all PB's data structures uses the same interface, that being the array interface. Most of the legacy code i'm ok with, some of it i'm updating to bring in some of the newer ideas explored in it's descendants. Haven't had much time to really chomp through it, but what I have been slowly working up the core routine with their support code. One such replacement routine was the ReDIM function, which is a lot more complicated than the DIM operation.
Dim is fairly dumb, it just creates some empty (formatted) space for the array. There's a little more to it than that, but conceptually that's about it. ReDIM on the other had has to rebuild the structure when the sizes are different. If you make an array larger, then it allocates some new memory and copies the old data across, the same goes for shrinking an array. It copies the existing data,not forgetting to deletes any excess. Which can be an easy way to leak resources if you're not careful. See-> Tips On Safe Resource Management:
The other thing i've been working on is the back end support for the linked lists. A link list is an optional extension to the array structure, so when a list is present the address resolver uses the structures internal current position. So far it's in a very bare bones state, due to how interdependent some of the back end actually is when dealing with lists. For example the list iterators are set up, but insertions don't work because that requires insertion management of the arrays size, hence the need for ReDIM.
But here's a general idea of where it was at last night..
PlayBasic Code:
Function DLL_ResizeArray(Size)Dim Cool(Size)For lp =0to Size
Cool(lp)=1+Rnd(10000)nextredim Cool(Size+10)EndFUnction COol()Type Dude2
xxxx,yyyy,zzzz
string$
endtypeFunction Dll_TestLIst(Size)Dim Z as dude2
Dim L as dude2 listFor lp=1to size
l=new dude2
l.xxxx=lp
print l.xxxx
nextEndFunction L.Dude2
I think it should be smoother sailing once all the little inter-dependencies have been ironed out.
PlayBasic To Dll - Macros make the world turn faster
The back end assembly is built with flat assembler which has some very impressive macro support, most of which is way beyond the scope of the project and my understanding of it . However, macro's can be a very powerful approach to programming, in particular in assembly, as they allows the user to dramatically reduce the lines of code, where a common behavior is wrapped as a macro and used throughout the project. It's a bit like a function, except the macros are parsed out by the assembler at compile time. So all the code is inlined.
So what i've been doing is moving more of the code in the base template into macros. You could pretty much make macro's for all the core operations, which would simplify PlayBasic to Dll's job by hiding how something works away in the template. The current build of the generator is pretty reactionary, so when it sees some known byte code pattern it spits out the same' template assembly chunk in reply. Such things could well be macro'd, allowing improvement to be make to the template and not only via the translator.
PlayBasic To Dll - Data Statements (January 22, 2014)
There's only a few tidbits left in the entire conversion felt to do and DATA statement support is one of them. Initially I wasn't going to bother with them, in favor of resource binding, but local data is probably easier for most people to wrap their heads around. There's a few gotcha's with exporting data to our DLL's though and that's visibility. In the VM data is global regardless of where it's defined, so the data heap exists outside of the program code. Now if you compile a PlayBasic source to DLL that has Data in it, now the question is who should have access to that data.
It's viable that the VM could retrieve any local data from the a bound dll and pull it into the global pool on the VM side, but then the dll's have to use the VM access. Which is about as painful as it sounds ! - I think i'd much rather that any data be local to it's dll, which makes a lot more sense in terms of encapsulation anyway. If the dll's shared data with the main program this could affect the users code in the VM or even in other bound dll's. Making it a slippery slope.
The DATA command set was re-written for the VM3 prototypes, but after looking over the replacement code, there's a few areas where it could be further improved. I think some of those changes would have to be made at compiler level to really see the benefits, but one thing that comes to mind would be replacing how the string heap is set up. If you have a lot of string data in a program and those strings share fragments/words, then there's some lost opportunities in terms of compression.
Ie.
Data"Hello World", "Hello World2"
The above data fragment would store two strings, since they're not a perfect match. But what it could do at build time is compare smaller fragments with the raw pool. So any matching runs would compress away, both still exist, it's just one is a part of the other.
Stuff like this doesn't make any significant difference to small programs, but as programs get bigger the amount of partial collisions increases.
PlayBasic To Dll - Function Calling Cont. (February 03, 2014)
Yesterday was get function calling done day, but things didn't turn out how I'd hoped. In fact spent almost 12 hours on it, only to run into brick wall after brick wall. Was initially going to wrap up some functionality in the library to handle an emulated VM stack, which I know would work and would be a perfectly fine solution, but it just doesn't sit well with me. My preferred solution is to use the system stack, but that's where we hit a few dramas... PB's internal calling convention has type casting wedged into it. This makes sense in the VM, since it reduces the number of VM ocodes needed to execute a new scope. But it means that any interface has to be data type aware when pushing or popping. Macros to the rescue !
The solution was to write a set of tiny macros that emulate the behavior of the VM in assembly. Since they're macros, the code is inlined into our exported dll code. So there's no calling a VM function to do the task for us. The up side is it's about as fast as you can get it, the down side is it adds a little extra bloat to the amount of code that's produced around a function and a function call, but personally it's not enough to worry about.
So far, the caller only supports passing primitives such as Integer/Floats & Strings in and out. Multiple returns (inside PB) are supported, but I haven't actually tested to see if they work correctly as yet. Pointer and Array passing isn't currently hooked up..
Some test code:
PlayBasic Code:
Function DLL_ExportMe(A,b#,s$)Dim Dude(100)Print"Calling Local PB function"
result=TestFunction(A,b#,s$,Dude())print"result="+str$(result)EndFunction result
Function TestFunction(A,b#,s$,Dude())print a
print b#
print s$
result=A+b#
print"array Status"ifgetarraystatus(Dude())printGetArrayElements(dude())endifEndFunction result
In this program we see two different types of functions. The first one DLL_ExportMe is our public / exported function. This function is visible to other programs that load this compiled DLL. Since it's a DLL function, it uses the STDCALL calling convention rules. The second function TestFunction (which is called from the first in this example) is a local function to the dll. This function is not visible externally and uses the PlayBasic's calling convention internally, so even if you have a pointer to this code, it's not callable externally.
EDIT #1 - testing multiple returns from functions
This seemed like it initially would work, but there was problem with the translator using a previously defined instruction in an unexpected way. Once that that was picked up it seems to work OK.
Recursion isn't currently supported though.
PlayBasic Code:
a=666
b#=123.456
s$="hello"
DLL_ExportMe(A,b#,s$)SyncwaitkeyFunction DLL_ExportMe(A,b#,s$)Dim Dude(100)Print"Calling Local PB function"
result=TestFunction(A,b#,s$,Dude())print"result="+str$(result)
result,result2=TestFunction(A,b#,s$,Dude())print"Multi returns"print"result="+str$(result)print"result2="+str$(result2)print"------------------------"EndFunction result
Function TestFunction(A,b#,s$,Dude())print a
print b#
print s$
result=A+b#
result2=result+1000print"array Status"ifgetarraystatus(Dude())printGetArrayElements(dude())endifEndFunction result,result2
PlayBasic To Dll - Range / ClipRange functions (February 21, 2014)
Added support for the integer Range instruction yesterday, you can't mix types at this point. For expressions that pass Range a mix of floats, it's probably going to be best solved by calling a function (the call size is smaller than my inlined code). Which is something I've been trying to avoid for maths functions where practical.
The ClipRange functions are borderline also, I suspect there's situations where the output code could be simplified more than calling a one size fits all function , like where the range terms are literal. If that occurs, it could output a compare block in it's place. Which would be faster at runtime !
ie.
Result=ClipRange(SomeValue, 100,200)
could become something like this (in BASIC styled logic)
Register=SomeValue
if Register<100 then Register =100
if Register>200 then Register =200
Result =Register
I could just take the easy way and cookie cut everything, but there's plenty of BASIC compilers that already do that..
PlayBasic To Dll - For/Next Problems
It's not all rosey though as one of the problems i've been running into recently, stems from the very nature of the byte code itself. The VM uses self modification in places, which is very handy for removing decisions from certain opcode pairs. But can make translating the logic back something of a pain.
For/Next loops are one of those areas, where the compiler produces all number of combinations at the start the loop sequence, but these opcodes are generic, there's nothing special about them and they appear in other constructs as well, so detecting exacting what's going on isn't turning out to be that easy.. The result of which, is sometimes the translator misses the FOR loop set up code, when that occurs the for/next doesn't work in the exported DLL. I'm hoping there will be some rainbow moment where I can figure out a bullet proof solution, but nothing coming to mind at this point.
There's a few solutions, worse case would be adding bread crumb opcodes to the byte code in order to help any translator(s) process down the line. Another approach would be change how the VM works and use add loop start opcode which would make it easier to translate and possibly at runtime for VM also. I kind of favoring the latter, but we'll see..
PlayBasic To Dll - GUI - Up And Running ! (March 28, 2014)
After a frustrating week of playing hard drive roulette, yesterday i've continued putting all the parts together. The build process is broken into two halves, there's the GUI side and translation engine side. Unlike other helper apps, these are two separate projects, which are combined using a slightly more customized version of the project loader/source merger. This version of the tool lets me wedge the Engine into the GUI easily and rips unwanted blocks of the code from the engine. The pre-processing tool works via looking for tokens inside comments. So the tokens mean nothing to PB, but the pre-processor tool just skims through looking token pairs and rips anything between them.
So the code (in this case a function) is included when engine when testing separately, but would be ripped when the engine is merged for inclusion into the GUI. I often use such approach to substitute methods in the final source codes for example. Once the engine source is built, I just include it into the GUI project and off we go. The GUI side holds the high level properties and just calls the conversion functions from the engine. To make the engine project output to the GUI rather than the odd print statement we just wrap up our own print function. So depending upon the context, the wrapped function either uses the GUI console or a print to screen.
When running the final program the user has to initially locate their PlayBasic.exe compiler file. This will generally be in your My Programs or Program Files folder it's different between OS's. I can sniff this out only when install in the most common location, but can't if you install to a custom location. To find the file go to 'SETTINGS->CONVERSION' and click the locate button. The dialog is set up to only accept one file. It's just matter of you finding it. Once the location has been set, it remembers this path.
To convert a PB source (PBA file) we go to FILE->CONVERT FILE, this pops a file locate dialog where all you do is select the file. PB2DLL will take over from there, there's is one little issue though and that's what your DLL name is going to be. Unfortunately, you can't create the DLL then rename it later, that won't work as there's naming dependency. So the name has to be given to the DLL up front. There's a few ways around the issue i guess, the DLL name could become the source files name. This is fine for those people that give their files meaningful names, but unfortunately a lot of people don't. One way around the problem would to be search through the compiled byte codes constant table for a constant called "PB2DLL_NAME$" or something similar, another would be to use a DLLNAME_ prefix in a function/psub declarations . Either way the programmer can then simply add the name string to their source code and then not have to enter a name during translation. Which would get old really fast if your constantly editing -> building -> repeat..
So anyway, it's up and running.. just matter of tweaking everything now.
PlayBasic To Dll - Pricing & Back End ( April 01, 2014)
PB2LL is something of a first around here, up until now all the helper tools have been freebies, PB2DLL is a commercial tool. The initial price will be for the early adopter version ! Which will be similar to what PlayBasic retail current costs, so somewhere between $20->$30 US. The final will most likely cost around $50.
One of the most frustrating things when releasing new software is all the secondary stuff that's required. It's not just the software you have to write, it's the ordering infrastructure it requires as well. Will probably go with shareit again, but contemplating just using PayPAL. Much like the DTAB ordering process, the KEY creation side of the ordering process won't be automated. You'll most likely order, then your registration will be sent to you once the order has processed locally. Which is required, since the key builder process is written in PB. Shouldn't be a drama really, as we're unlikely to see more than a few sales.
So today's little chore has been working on the activation system library which is just matter of dragging some of the code around from older libs, was a little painfully setting it up, but the process seems to be working. With other tools i've build pre-processing tools to do the entire process, which is tempting, but again can't really see any great demand..
PlayBasic To DLL (Convert PlayBasic to Machine Code DLL's)
I've been quietly working on two sides of the tool in parallel, the translator tool, as well as making updates to PB so they can integrate side by side. It's pretty slow going as some things are added quicker than others, in particular commands that aren't actually commands in the legacy VM, which need to be wrapped in order to create an external interface. Most of the core operations work like this.
The current edition of the translator supports around 160->170 core commands as well as Integer, Float, String, Pointer , Arrays (int, float, string, used defined typed) and core logic. The great thing about using the PlayBasic compiler to build the byte code, is the resulting code has lots of redundancies removed for us. Enabling us to export cleaner & faster executing assembly.
The assembly generation routines are a cross between cookie cutter and smart at this point. It all depends upon the operation, the current generator can only optimize output code for extension speed, when it notices friendly sequences. The excitingly thing about that, is the machine code is already routinely faster than our competitors.
Here's a few tidbits from the Work In Progress blogs.
PlayBasic To Dll - One Stop Shop - (August 20, 2013)
It's 5:09am and the tool finally has it's first taste of automation. Previously, I had to compile the PlayBasic code to byte code, then copy the output object code to the test folder, run the convertor, cut'n'paste the resulting source code fragments into the dll template, assemble and repeat.. over and over... Which gets old real fast ! So the goal tonight has been to get the program to a point where I can point it at a PlayBasic source file (PBA file) and it'll call the compiler, build the byte code and do all the conversions itself. There's a couple of hacks for the time being, since the compiler will need a specific mode for this stuff, But all in all, it's pretty painless.
The build speed is pretty good, even though the test sources are very simple, the actual conversion and assembly stage is consistently executing in about 50 milliseconds (or less). The conversion engine only supports a single PlayBasic source file at this time, it doesn't support includes (couldn't be bothered at this stage).. The resulting DLL will have all the code in the source in it, even if it's not in use. So ideally you'd use it to compile stand alone include files that perform some brute force task. The tool expects the functions you want to be made visible (exported) to have "DLL_" at the start of the function name. The tool rips this off, it's just a way to ID what functions you wish to export, without having to change the PlayBasic compiler dramatically.
Once the DLL is built, you can not only link this to your application but bind it to be executed from memory (doesn't need to be exported to disc). Size wise the resulting DLL's are pretty small, weighing in about 5K for the output of the following.
PlayBasic Code:
Function DLL_JumpTableTest(ptr); integer for next step loopFor lp =0to100step10pokeint ptr,lp
ptr+=4nextprint lp
EndFunctionFunction DLL_ForLoopTest(ptr)
StepY=10
StepX=10
Counter=0; integer for next step loopFor ylp =0to100step StepY
For xlp =0to100step StepX
Counter++nextnextFor ylp =100to0step-StepY
For xlp =100to0step-StepX
Counter++nextnext
a=100
b=200swap a,b
print Counter
EndFUnctionFunction DLL_JumpTableTest_BIG(a); integer for 'next with literal for lp=0to100
a=b
next; Integer for next with variable/register end valuefor lp=0to B
a=b
next; Nested integer for / next loopFor xlp=A*B to100For ylp =0to xlp*100
Stuff=xlp
nextnexton a goto label0,label1,label2,label3
a=999goto done
Label0:
a=0goto done
Label1:
a=111goto done
Label2:
a=222goto done
Label3:
a=333goto done
Label4:
a=444
Done:
a=addtest(a,40000)EndFunctionPsub AddTest(A,B)
result=a+B
EndPsub result
PlayBasic To Dll - Fractals anyone ? - ( August 24, 2013)
Attached you'll find a nice shiny example of what the DLL convertor is able to do today in the form of fractal render. I've modified the code slightly to work around a few functions that aren't supported as yet (the original code is in the source code board), but the result is a pleasant 18->19 times the performance improvement. The demo is drawing 640*480 pixels with potentially 200 square roots per pixel, So yeah.... that's a lot of work for the runtime to try and brute force.
The convertor currently has a hand full of opt's it can make when translating / exporting code, but it has no real awareness of the register management, so it's hitting the variable heap a lot more than what it should (extra memory accesses). Even without bust a gut though, it's working pretty well for now on my 8 year old athlon system..
Made a few register tweaks before tea tonight, then ported the bench mark code to a couple of competitors. No surprises the PlayBasic version runs 2.8 times faster than one, 4.5 times faster than the other... Oh dear, how embarrassing that must be for them...
PlayBasic To Dll - Strings -(August 28, 2013)
Been working on getting strings working most of the afternoon only to run into a strange crash when two string are added together. On inspection one string was legal (Hello World) and the other was null for some reason.. It's funny how your mind focuses in on a segment of code of you think is the problem, only to find the issue a few lines above it. Turned out the function initialization was killing the string buffer. Once corrected, it worked as expected.
So far I've only got a handful of core operators hooked up, it's just a matter of joining the dots. Something that is interesting is the when we start talking string management, we actually get less benefit from translating to machine code than what you might think. Unfortunately there's this idea out there that machine language is the golden bullet, but really it isn't and never was. In regards to strings.. well, there's no 'instruction set' in your CPU for doing string operations, everything is just reading and writing arrays of bytes. So if we add two big strings together, then regardless of how this operation was called, be it from machine code or the VM, the string joining operation is taking 99.9999% of execution time here.
Strings are a notorious bottle neck of programming languages, knowing that, the PlayBasic string engine is a very optimal solution, every effort has gone into making it as quick as possible. That's why I write string processing apps in it..
Here's a little something to bend your reality.. The function joins Hello World together 101 times. There's two version of the test, exported DLL version and the VM function. Bellow is clip from the current test code. So basically PB2DLL is pointed at this source and we get a nice shiny dll version couple of seconds later..
Yep, obviously the machine code version is going to be faster, but the interesting thing is by how much ? It'd be easy to assume it's going to be 5->10 times faster, where it's actually only around 35% faster.. Why, because it's spending most of it's time copying characters, not executing VM opcodes.
To put that in some real world perspective, DLL version is 12 times faster than one competitor and 18 times faster than another.
Getting this working has been a detoured chain of events. The first problem was the disassembler didn't support most of the pointer opcodes I needed, so the first port of call was adding that functionality, just so I could translate it back into assembly. After adding the decoder, it's here we notice there extra additions in the array field writes with type pointers. Could have ignored this, as an extra opcode in output might not like sound like much waste here and there, but if that code is sitting in some brute force loop, then it's throwing away 1/3 of the operations performance for nothing. This wastefulness would then be translated to the machine code DLL too, so it'd just be extra unwanted wasted cycles having a negative impact when the code is within a loop.
The only way to solve such problems is fire up PlayBasic and take a look at what it generates in particular situations. For some reason it was adding the structure displacement offset to the temp pointer register, adding the array offset, then doing the write. When all it should need is the adding the array offset, then do the write since the writes opcodes support displacement. So the offset is virtually free. Moreover, it didn't support literal array indexes, which can be pre-computed at compile time and represented as one displacement. So that was yesterdays little chore. The results are as expected as it's 30% quicker. I suspect there might be a few more situations like that hidden away also.
Tonight's mini session has been all about hooking up the assembly generation side, which gone relatively well really. Meaning it can produce working code from the following.
So it's getting to pretty familiar level of functionality, there's of course plenty of no no's just waiting to trap you. The main one that comes to mind, would be the lack of auto casting in some translated operations. Like if you have a FOR/NEXT loop, then it currently only supports Integer loop counters, where in PB you can have Integer or floating point loop counters. Same with things like function parameters. If a function expects an integer and you pass it a float, you can get away with this in PB, since the runtime is recasting the parameter on demand, but the translator tool doesn't currently support this. It will.. just not today.
(To get the most up-to-date info, you'll have to read the complete blog on our forums)
With the completion of the V1.64O retail upgrade, our focus has shifted onto the way forward for the classic PlayBasic platform. There's a number of options with equally as many positives and negatives. One of those options was building a translation tool to convert PlayBasic byte code to machine code in the form of a DLL.
Such a tool would mean PlayBasic programmers can create plug in's that perform any brute force processing required. The programmer could then bind (and execute) the DLL functions directly from PB resource in memory, never needing to be extracted them to disc.
The project is already up and running, you can following it's development on our forums. You'll need to sign up & log in to do so. Bellow is a few snippets from the blog.
Work In Progress (The Dissassembler Short Cut)
This idea has been floating around for a while now, there's a number of older tech demos that convert VM instructions into x86 machine code at run time (JIT). The process is relatively easy, it's just those programs are doing it manually (in PB code). So it'd read the byte code and drop out an equivalent x86 code into some executable memory, then call when done as a viability test. In principal you could do the same to export binary as dll, but an easier option would be to spit out raw assembly an pass through an assembler and hey presto job done.
Ok, so what does this have to do with the dissassembly tool written last year ? - Well, that program runs through PB byte code and converts it back into a semi readable text file for internal debugging. The result isn't pretty, but the same raw process could be used to drop x86 assembly out in place of the command simulations. So it should be possible, Relatively easy actually to get it to output at least simple operational code. It'd probably required the user PlayBasic code be set up in particular way, so the export and ID what functions are to be exported.
To export a function, the function name needs DLL_ like so,
PlayBasic Code:
Function DLL_FillMemoryInt(StartPtr,Count,ThisValue)dopokeint StartPtr,ThisValue
StartPtr+=4decloop Count
EndFunction
The nice thing about using PlayBasic compiler as the initial stage of the process (apart from saving me work), is that we get whatever benefits it applies to the input source in terms of optimizations. The output code wouldn't be comparable to hand written equivalent but obviously it'd give you some brute force muscle.
We've tested the above running in PlayBasic V1.64O compared to an equivalent assembly function and the classic VM takes about 90 milliseconds to poke (800*600) pixels and the Dll function takes about 3.5 milliseconds. So in this case, it's more like 25 times faster than the classic VM. Not only that, the compiled DLL is also 25% faster than one competitor and about 10 times faster than another.
Programming has enough annoying problems as it is, let alone trying to covert legacy source code around between languages. Bizarrely the problem hasn't really improved that much since when I started programming back in 1982, at least then there was a good reason, like.. you know... assembly - Today, there's lots of conversion tools between main stream languages, but there's just so many languages out there, lots of legacy code just ends up sitting gathering dust.
I dunno about other programmers, but I always try to hang onto my legacy source codes in one big retro collection. The oldest stuff dates back to the 90's (lost my 8bit stuff ;( ). The code is mostly AMOS and 68k Assembly from the Amiga days. The assembly code isn't much use, but a lot of legacy BASIC code can be. The trouble with languages like AMOS, is that the source code files aren't raw text, they're stored in a tokenized format. So you either need AMOS to view them, or a conversion tool.
Converting AMOS / AMOSPRO source code to PlayBasic
Those with a long memories might recall I've had a few tinkers with writing translation programs over the years, even getting a 'mock up' translator to produce actually working conversion of some simple AMOS programs (and a few other dialects) programs. While the Amos convertor worked (to a point) it was basically a brute force solution, since that translator was working at string level making it very hit and miss.
Normally when parsing source code, you'd translate the text stream into a tokenized state, then decipher what's going on from the tokens. But for AMOS programs, it makes sense to load the tokenized code and decode that directly. I'd looked at doing this before and got a fair way into it, only to find a tool that already convert AMOS V1.2V1.3 codes, but not AMOS PRO. So the decision was made to drag out my older attempt and cut'n'pasted that into new project from it's ashes.
Only took a few hours to get it a reasonable translation of a real world binary Amos program working. But what you have to remember, is that AMOS is huge language and I've absolutely NO intention of building a one stop shop convertor. Rather, the objective was to make the tool spit out a reasonable translation that any programmer can then use as a basis for the PlayBasic version. Meaning the program LOGIC should be in tact, but of course there's no easy way of mapping the entire AMOS command sets to PlayBasic without writing an abstraction layer. Which is possible for simple programs (since that's how the previous test worked) but somewhat unrealistic for medium to large AMOS programs.
What's Supported ?
The tool currently supports all versions of AMOS including 3rd party extensions. However, not all the Amos command sets are mapped though, but these can be defined externally by the user. This can mean a little messing around initially, in particular if the code you want to covert uses extensions heavily, but unfortunately Amos source files don't include such mappings in them, so we can't do it automatically.
Beyond just loading the Amos sources, the program applies a number of translations to the original input code prior to saving the PlayBasic styled version. Ranging from the simple declarations, converts procedures to functions (supports encrypted procedures), variable name collisions and any number of core Memory.Math/String function & operator level conversions. Now depending upon what command sets the original AMOS code uses, will dictate how 'ready to run' the translation is. But it's highly unlikely that programs will run first go however.
One of the nice things about using a binary format for your source code, is that AMOS programs often had data hunks attached to them. These hunks range from SPRITES, ICONS, PICTURES, SAMPLES, MUSIC etc etc. The user can exported them (optionally) with the translated source code.
For More Information: See Amos To PlayBasic Work in progress thread for downloads.
Rather late(or very early) Easter game
by Monkeybot
This game was originally written for the 2011 Easter competition, but the author (monkeybot) didn't make the deadline. In this game the player must safely guide the egg through the tunnel without touching the walls, while the eggs spinning. So the slightest inaccurate mouse movement and you're dead.
It's a pity he didn't get a chance to complete this game, as while it might seem simple on the surface, it's anything but. Well worth a look since he's generously included the source code.
Minigame - Feedback wanted
by Basil99
This is a minigame concept we plan to add to a bigger project. I'm personally not sure this is a playable thing, as for me its a bit boring. Anyway, community feedback is wanted. Plz, leave your opinion about this.
Idea is a following - a big picture is spit into square pieces, like 4x5. Then, some pieces are randomly removed and remained structure is falling down. Your purpose is to restore original pic by catchin falling block by moving "basement". Well, sounds unclear, but may be demo helps
Play Pic2Map (Conversion Tool)
by Kevin Picone
It's a tool to assist PlayBasic users, convert Image into something that can be drawn as a map. The user loads the image into the program and it exports a tile map, and an image containing the unique blocks used.
Donkey Kong Country 2 Remake (2D side scroller engine WIP)
by LemonWizard
This is one of those ambitious remake projects that pops up from time to time, basically the concept here is to create a Donkey Kong Country 2 Styled scrolling platform engine.
The author (LemonWizard) is taking a tools based approach to creating his solution, where he's starting out working on a editing tool that has play back functionality built in. This allows him to build components and test them as he's working.
If you're interested there's plently of Work in Progres chat/ screen shots / videos and source code snippets posted in the WIP thread.
The author (LemonWizard) is taking a tools based approach to creating his solution, where he's starting out working on a editing tool that has play back functionality built in. This allows him to build components and test them as he's working.
when comparing the byte code state being compiles, there's an odd similarity in where the errors occur in them, in particular in the map that flags byte code as data and to be skipped.
