Alternative parser solution

genius257

So i don't know if this is helpful to the EasyCodeIt, but i just wanted to put it on here.
I've tried to implement a AutoIt3 parser in AutoIt3 for some time, always drowning in the complexity and lack of complete valid syntax spec.

I've tried tackling it again, but this time via a simplified format that can produce the parser code itself for me via PEGjs/PeggyJs: https://github.com/genius257/autoit3-pegjs/blob/main/src/autoit3.pegjs

I hope that the grammar file I'm making COULD help save some time when developing the parser for EasyCodeIt?

When it gets more production ready i will post more about it on the official AutoIt forum, if anyone cares 😀

TheDcoder

That's awesome, thanks for sharing. I had a look and it is definitely something I'd be interested in, for now I have starred it on GitHub and I will revisit it soon, perhaps by the end of this weekend 🙂

TheDcoder

I just tested your parser in the browser and it is working, and it appears to be more feature complete than the one I've written, I really appreciate you working on this, it has a lot of potential!

If I find that the current bison approach is getting too tedious perhaps I can adapt your parser and compile it into web-assembly and use that for parsing 😁

genius257

TheDcoder

Happy to hear 🙂, let me know if you do 😉

I do need more testers than me, to expose edge cases and undocumented features, that might take me years to find on my own, if at all.

for example:

Click to reveal Click to hide

Redim accepts multiple variables separated by commas, but that is not documented

I am currently in the process of creating a vscode extension that uses it for intellisense stuff, but i also have crazy plans of making AutoIt3 being able to interpret, execute and compile au3 code at run-time (in the far future).

TheDcoder

genius257 For sure. Cool find about ReDim, but it might be a quirk that was unintentionally developed in the parser because Local and Global accepts commas naturally. That or the developers forgot to document it!

genius257 I am currently in the process of creating a vscode extension that uses it for intellisense stuff

Great, best of luck!

genius257 but i also have crazy plans of making AutoIt3 being able to interpret, execute and compile au3 code at run-time (in the far future).

Something like the Execute function? I used it on of my programs many years ago and it can definitely work with expressions and even variable declarations, but I didn't try any complex statements.

genius257

TheDcoder Something like the Execute function? I used it on of my programs many years ago and it can definitely work with expressions and even variable declarations, but I didn't try any complex statements.

Yes and no. It is true that the execute function can run strings as code, but that is only for a single expression/statement. Also Execute does not allow assigning values to variables. For that Assign is needed.

Examples of features that you cannot currently do:

functions cannot be defined at run-time, like in JavaScript for example.
constants cannot be defined via the Assign function.
you cannot debug at run-time, like most other languages.

These things could be solved with a AutoIt3 parser and executor.

Then there is the compiler aspect:

turning AutoIt3 code into machine code at run-time could improve problematic things like big loops speeds
the above COULD open the possibility of thread support in AutoIt3
producing machine code could make it possible to produce a self-compiling compiler. This COULD have many benefits, including making a open source version of AutoIt3 that can have new features added over time (a long term project i have named AutoIt3+)

Now how much of this will end up being achieved, time will tell 🙂

TheDcoder

genius257 Why does your "AutoIt3+" project sound eerily similar to my EasyCodeIt project? 😛

Given all of your goals I think your looking at a complete language with its own tooling, or did I miss something?

producing machine code

Aside from being hard to write and maintain, you'll completely miss out on alternative platforms like ARM and RISC, the former is gaining popularity very rapidly and has already taken over in numbers if you could non-PC devices.

The new apple hardware contains a new power-efficient M1 processor which was designed by them in house, and it uses the ARM architecture, that's the best example we have for a PC use case right now. And don't forget the people who would want to use their Raspberry Pies for everything 😃

Perhaps we can alleviate this issue by producing an intermediate instruction set and using something like LLVM to recompile it into many different machine codes.

functions cannot be defined at run-time, like in JavaScript for example.

I'd like to mention a pedantic detail here, you aren't really creating new functions at run-time, you are just creating new instances of the same function which runs in a different context!

Unless you mean loading entirely new functions from strings using eval!

genius257

TheDcoder Why does your "AutoIt3+" project sound eerily similar to my EasyCodeIt project? 😛

I admit that there are similarities 😉

TheDcoder Given all of your goals I think your looking at a complete language with its own tooling, or did I miss something?

Eventually yes, but I plan on taking it step by step, learning from each one.
AutoIt3+ is the final project, and i expect it to take many years to get there 😬

AutoIt3 parser
A executor in AutoIt3 for the parser output
First version of AutoIt3+, written in AutoIt3
Experimenting with machine code for the executor
A compiler
A stand alone open source AutoIt3+ executable

Now this is very rough and some steps are missing I'm sure 😛

TheDcoder Aside from being hard to write and maintain, you'll completely miss out on alternative platforms like ARM and RISC, the former is gaining popularity very rapidly and has already taken over in numbers if you could non-PC devices.

The new apple hardware contains a new power-efficient M1 processor which was designed by them in house, and it uses the ARM architecture, that's the best example we have for a PC use case right now. And don't forget the people who would want to use their Raspberry Pies for everything 😃

Here is where EasyCodeIt and AutoIt3+ goals differ.
I have no plans to support other systems than Windows.
This is simply because of backwards compatibility and the important fact that DllCall would not work anymore.
It is possible to have different code called for alternative systems, but personally imagining all my projects supporting that gives me a headache 😵‍💫

TheDcoder Perhaps we can alleviate this issue by producing an intermediate instruction set and using something like LLVM to recompile it into many different machine codes.

I personally do not plan on doing this, but AutoIt3+.
It is a good idea. Someone might be able to make a fork that does this in the future, if my code end up being written in a matter that does not require an entire re-write.

TheDcoder I'd like to mention a pedantic detail here, you aren't really creating new functions at run-time, you are just creating new instances of the same function which runs in a different context!

Unless you mean loading entirely new functions from strings using eval!

Well eval is one option, but JS also support making functions at run-time with new Function(). Example:

const sum = new Function('a', 'b', 'return a + b');

console.log(sum(2, 6));
// expected output: 8

TheDcoder

genius257 Well eval is one option, but JS also support making functions at run-time with new Function(). Example:

I didn't know about this feature, I guess you learn something new every week 😆

But the premise is still the same if you ask me, the Function constructor is just a wrapper around eval from the looks of it.

AutoIt3+ is the final project, and i expect it to take many years to get there 😬

It's a cool plan, you're going to bootstrap AutoIt3+ with AutoIt3! I think the biggest hurdle you are going to face is the compiler to native code, you will have to take care a lot of stuff since AutoIt is a high-level language.

This is simply because of backwards compatibility and the important fact that DllCall would not work anymore.

But you can easily provide backward compatibility as a different mode, and nothing is stopping you from making a DllCall function in Windows as a polyfill using some other generic shared library calling interface, Linux would need something too 🙂

That's how I'm planning to provide back-compat in ECI.

By the way, I'm curious about the motivation behind your idea, why write AutoIt in AutoIt?

genius257

TheDcoder But the premise is still the same if you ask me, the Function constructor is just a wrapper around eval from the looks of it.

mostly yes, but it is still a way to import new JS code into a program at runtime. My issue with AutoIt3 is that Execute only allows this partially, and a whole codebase is needed to mimic the full experience.

TheDcoder It's a cool plan, you're going to bootstrap AutoIt3+ with AutoIt3! I think the biggest hurdle you are going to face is the compiler to native code, you will have to take care a lot of stuff since AutoIt is a high-level language.

Indeed 🙂 I may never get to that point, but maybe one day 😃

TheDcoder But you can easily provide backward compatibility as a different mode, and nothing is stopping you from making a DllCall function in Windows as a polyfill using some other generic shared library calling interface, Linux would need something too 🙂

That's how I'm planning to provide back-compat in ECI.

Hmmm... i guess that is correct.
I guess I might end up doing something like that when i get to that stage of the process. I'm sure you will make it a feature request if i forget 😛

TheDcoder By the way, I'm curious about the motivation behind your idea, why write AutoIt in AutoIt?

Well mostly it's a shortcut for supporting the correct AutoIt3 behavior. AutoIt3 does some strange things sometimes that can be difficult to detect (COM behavior, custom window message proc, legacy windows support and expected order of execution with asynchronous and synchronous code just to name a few).

I also think it lowers the bar for reading and understanding the code (somewhat).

And my final reason is that i like to squeeze functionality out of AutoIt3 that does not seem possible initially (i think, any way), it is my hobby language after all 😛

TheDcoder

Well mostly it's a shortcut for supporting the correct AutoIt3 behavior. AutoIt3 does some strange things sometimes that can be difficult to detect (COM behavior, custom window message proc, legacy windows support and expected order of execution with asynchronous and synchronous code just to name a few).

I see, never heard or stumbled across this issues before personally, probably because I don't use these features very much.

What do you mean "asynchronous" code by the way? There's no such thing in AutoIt AFAIK. The closest thing I can think of are functions which are called back by GUI events or AdLib.

genius257

TheDcoder I see, never heard or stumbled across this issues before personally, probably because I don't use these features very much.

That's just it. They're not widely used, but may end up causing major code refactors down the line 😟

TheDcoder What do you mean "asynchronous" code by the way? There's no such thing in AutoIt AFAIK. The closest thing I can think of are functions which are called back by GUI events or AdLib.

Well asynchronous in the sense that they may run at non determined intervals based on external triggers.
Things like: GUI events, DLL callbacks, AdLib and COM object events.

Finding out what takes precedence may not seem to bad at first, but I'm sure it is not simple to replicate in all cases.
This may not be a huge issue, but i am sure it could create major problems for some bigger GUI/COM projects, where a change in the order of execution could make the program crash or cause unexpected behavior.