EasyCodeIt Progress: Flex & Bison parser

TheDcoder

Hi everyone,

I already posted an update yesterday where I mentioned that I have started work on the new parser which uses Flex and Bison to generate C code for parsing.

So far the experience has been very nice, I am able to test and develop things very fast compared to doing it all on my own.

I have completed the initial version of the tokenizer/scanner/lexer, it should be able to scan all tokens which could possibly occur in a valid script 🙂

%option noyywrap nodefault

%{
#include <stdlib.h>
#include <string.h>

char *token_str = NULL;
size_t token_len = 0;

size_t comment_level = 0;

//#define YY_USER_ACTION free(token_str); token_str = strndup(yytext, token_len = yyleng);
#define YY_USER_ACTION token_str = yytext; token_len = yyleng;

enum { // Token Types
	UNKNOWN = 420,
	WS,
	COMMENT, DIRECTIVE,
	NUMBER, STRING, BOOL,
	WORD,
	MACRO, VARIABLE,
	OPERATOR, BRACKET, DOT, COMMA,
};

%}

%x ML_COMMENT

WS	[ \t\r\n]+
NL	\r?\n?

DIGIT	[0-9]
XDIGIT	[0-9A-Fa-f]

QUOTE	[\'\"]

%%

 /* Whitespace */
{WS}	;

 /* Comment */
<INITIAL,ML_COMMENT>"#cs"|"#comment-start"{WS}	{BEGIN ML_COMMENT; ++comment_level;};
<ML_COMMENT>"#ce"|"#comment-end"	if (--comment_level == 0) BEGIN INITIAL;
<ML_COMMENT>(?s:.)	;
[;#][^\r\n]*	return *yytext == ';' ? COMMENT : DIRECTIVE;

 /* Number */
{DIGIT}+(\.{DIGIT}+(e{DIGIT}+)?)?	return NUMBER;
0[xX]{XDIGIT}+	return NUMBER;

 /* String */
\"[^\n\"]*\"	return STRING;
\'[^\n\']*\'	return STRING;

 /* Bool */
(?i:"True"|"False")	return BOOL;

 /* Word */
[A-Za-z][A-Za-z0-9]*	return WORD;

 /* Macro or Variable */
[@$][A-Za-z][A-Za-z0-9]*	return *yytext == '@' ? MACRO : VARIABLE;

 /* Operator */
[+\-*/^&=<>?:]	return OPERATOR;

 /* Misc */
[[\]()]	return BRACKET;
\.		return DOT;
\,		return COMMA;

 /* Catch-all for everything else */
.	return UNKNOWN;

%%

#include <stddef.h>
#include <stdio.h>

static void print_token(char *str, size_t len, int type) {
	puts("---### TOKEN ###---");
	char *token_type;
	switch (type) {
		case UNKNOWN:
			token_type = "Unknown";
			break;
		case WS:
			token_type = "Whitespace";
			break;
		case COMMENT:
			token_type = "Comment";
			break;
		case DIRECTIVE:
			token_type = "Directive";
			break;
		case NUMBER:
			token_type = "Number";
			break;
		case STRING:
			token_type = "String";
			break;
		case BOOL:
			token_type = "Boolean";
			break;
		case WORD:
			token_type = "Word";
			break;
		case MACRO:
			token_type = "Macro";
			break;
		case VARIABLE:
			token_type = "Variable";
			break;
		case OPERATOR:
			token_type = "Operator";
			break;
		case BRACKET:
			token_type = "Bracket";
			break;
		case DOT:
			token_type = "Dot";
			break;
		case COMMA:
			token_type = "Comma";
			break;
		default:
			token_type = "Unnamed";
			break;
	}
	fputs("Type: ", stdout);
	puts(token_type);
	fputs("Data: ", stdout);
	puts(str);
}

int main(void) {
	int type;
	for (;;) {
		type = yylex();
		if (!type) break;
		print_token(token_str, token_len, type);
	}
	return 0;
}

I passed this "script" as the input:

#cs
	#cs
		nested comment
	#ce
#ce

; Calm Mints

42
0xDEADBEEF
3.14
6.02214076e23

"Hell'o World"
'"To C or not to C"'

True
true
fAlSe
False
tRUe

@TheTruth
$TheFact

-((1 + 2) / 3) * 4

And here is the output:

---### TOKEN ###---
Type: Comment
Data: ; Calm Mints
---### TOKEN ###---
Type: Number
Data: 42
---### TOKEN ###---
Type: Number
Data: 0xDEADBEEF
---### TOKEN ###---
Type: Number
Data: 3.14
---### TOKEN ###---
Type: Number
Data: 6.02214076e23
---### TOKEN ###---
Type: String
Data: "Hell'o World"
---### TOKEN ###---
Type: String
Data: '"To C or not to C"'
---### TOKEN ###---
Type: Boolean
Data: True
---### TOKEN ###---
Type: Boolean
Data: true
---### TOKEN ###---
Type: Boolean
Data: fAlSe
---### TOKEN ###---
Type: Boolean
Data: False
---### TOKEN ###---
Type: Boolean
Data: tRUe
---### TOKEN ###---
Type: Macro
Data: @TheTruth
---### TOKEN ###---
Type: Variable
Data: $TheFact
---### TOKEN ###---
Type: Operator
Data: -
---### TOKEN ###---
Type: Bracket
Data: (
---### TOKEN ###---
Type: Bracket
Data: (
---### TOKEN ###---
Type: Number
Data: 1
---### TOKEN ###---
Type: Operator
Data: +
---### TOKEN ###---
Type: Number
Data: 2
---### TOKEN ###---
Type: Bracket
Data: )
---### TOKEN ###---
Type: Operator
Data: /
---### TOKEN ###---
Type: Number
Data: 3
---### TOKEN ###---
Type: Bracket
Data: )
---### TOKEN ###---
Type: Operator
Data: *
---### TOKEN ###---
Type: Number
Data: 4

Which is very similar if not exactly the same as what our old parser would have produced 🙂

The next step is to add support for including files (#include <...>) and then I will start work on the syntactic analysis, which is the actual "parsing" to convert list of words into actions. Hopefully I won't get held up by last time due to all the fatigue, using these tools should speed up the process significantly.

Stay tuned for more updates this week!

TheDcoder

I got busy with work again, but luckily I didn't take another month long break 😛

I implemented support for #include and #include-once in the lexer, this means the lexer part is more or less done. Here is the new code:

// lexer.l
%option batch noyywrap nounput nodefault yylineno

%{
/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2021 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */

#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

struct BufferStack {
	struct BufferStack *prev;
	char *code;
	YY_BUFFER_STATE state;
	size_t line;
	char *file;
} *lex_buffer = NULL;

static char *token_str = NULL;
static size_t token_len = 0;
static size_t comment_level = 0;

#define YY_USER_ACTION token_str = yytext; token_len = yyleng;

enum { // Token Types
	UNKNOWN = 420,
	WS,
	COMMENT, DIRECTIVE,
	NUMBER, STRING, BOOL,
	WORD,
	MACRO, VARIABLE,
	OPERATOR, BRACKET, DOT, COMMA,
};

#include "parser.h"

static source_reader read_file;

static bool push_file(char *file);
static bool pop_file(void);

%}

%x INCLUDE
%x DIRECTIVE_LINE
%x ML_COMMENT

WS	[ \t\r\n]+
NL	\r?\n?

DIGIT	[0-9]
XDIGIT	[0-9A-Fa-f]

QUOTE	[\'\"]

%%

 /* Include File */
 /* TODO: #include-once */
^"#include"{WS}[\"<]	BEGIN INCLUDE;
<INCLUDE>[^\n\">]+	{
	int c;
	while((c = input()) && c != '\n') /* Eat up any leftover junk in the include line */;
	++yylineno;
	push_file(yytext);
	BEGIN INITIAL;
}
<INCLUDE>.|\n	/* Ignore bad include line */;
"#include-once"	{
	/* Add current file to "include once" list*/
	read_file(lex_buffer->file, NULL, true);
}

 /* Whitespace */
{WS}	;

 /* Directive */
"#"	{BEGIN DIRECTIVE_LINE; yymore();};
<DIRECTIVE_LINE>.+	{BEGIN INITIAL; return DIRECTIVE;};

 /* Comment */
<INITIAL,ML_COMMENT>"#cs"|"#comment-start"{WS}	{BEGIN ML_COMMENT; ++comment_level;};
<ML_COMMENT>"#ce"|"#comment-end"	if (--comment_level == 0) BEGIN INITIAL;
<ML_COMMENT>(?s:.)	;
;[^\r\n]*	return COMMENT;

 /* Number */
{DIGIT}+(\.{DIGIT}+(e{DIGIT}+)?)?	return NUMBER;
0[xX]{XDIGIT}+	return NUMBER;

 /* String */
\"[^\n\"]*\"	return STRING;
\'[^\n\']*\'	return STRING;

 /* Bool */
(?i:"True"|"False")	return BOOL;

 /* Word */
[A-Za-z][A-Za-z0-9]*	return WORD;

 /* Macro or Variable */
[@$][A-Za-z][A-Za-z0-9]*	return *yytext == '$' ? VARIABLE : MACRO;

 /* Operator */
 /* IDEA: Make a parser mode where the character will be returned literally */
[+\-*/^&=<>?:]	return OPERATOR;

 /* Misc */
[[\]()]	return BRACKET;
\.		return DOT;
\,		return COMMA;

 /* Pop file and terminate if top-level */
<<EOF>>	if(!pop_file()) yyterminate();

 /* Catch-all for everything else */
.	return UNKNOWN;

%%

#include <stddef.h>
#include <stdio.h>

static void print_token(char *str, int type) {
	puts("---### TOKEN ###---");
	char *token_type;
	switch (type) {
		case UNKNOWN:
			token_type = "Unknown";
			break;
		case WS:
			token_type = "Whitespace";
			break;
		case COMMENT:
			token_type = "Comment";
			break;
		case DIRECTIVE:
			token_type = "Directive";
			break;
		case NUMBER:
			token_type = "Number";
			break;
		case STRING:
			token_type = "String";
			break;
		case BOOL:
			token_type = "Boolean";
			break;
		case WORD:
			token_type = "Word";
			break;
		case MACRO:
			token_type = "Macro";
			break;
		case VARIABLE:
			token_type = "Variable";
			break;
		case OPERATOR:
			token_type = "Operator";
			break;
		case BRACKET:
			token_type = "Bracket";
			break;
		case DOT:
			token_type = "Dot";
			break;
		case COMMA:
			token_type = "Comma";
			break;
		default:
			token_type = "Unnamed";
			break;
	}
	fputs("Type: ", stdout);
	puts(token_type);
	fputs("Data: ", stdout);
	puts(str);
}

static void scan() {
	int type;
	for (;;) {
		type = yylex();
		if (!type) break;
		print_token(token_str, type);
	}
}

bool parse(char *file, source_reader read_func) {
	read_file = read_func;
	push_file(file);
	scan();
	return true;
}

static bool push_file(char *file) {
	size_t code_len;
	file = strdup(file);
	if (!file) return false;
	char *code = read_file(file, &code_len, false);
	if (!code) {
		free(file);
		return false;
	}
	struct BufferStack *new_buffer = malloc(sizeof *new_buffer);
	if (!new_buffer) return false;
	if (lex_buffer) lex_buffer->line = yylineno;
	*new_buffer = (struct BufferStack){
		.prev = lex_buffer,
		.code = code,
		.state = yy_scan_buffer(code, code_len + 2),
		.file = file,
		.line = yylineno = 1,
	};
	lex_buffer = new_buffer;
	yy_switch_to_buffer(lex_buffer->state);
	return true;
}

static bool pop_file() {
	struct BufferStack *prev_buffer = lex_buffer->prev;
	yy_delete_buffer(lex_buffer->state);
	free(lex_buffer->code);
	free(lex_buffer->file);
	free(lex_buffer);
	lex_buffer = prev_buffer;
	if (!lex_buffer) return false;
	yylineno = lex_buffer->line;
	yy_switch_to_buffer(lex_buffer->state);
	return true;
}

// eci.c
/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2020 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */
 
static char *provide_code(char *file, size_t *size, bool once) {
	static char **once_list = NULL;
	static size_t once_list_len = 0;
	if (once) {
		// Add the file to "include once" list
		if (!once_list) {
			once_list = malloc(sizeof(char *));
			if (!once_list) return NULL;
			++once_list_len;
		} else {
			char **expanded_list = realloc(once_list, sizeof(char *) * ++once_list_len);
			if (!expanded_list) {
				--once_list_len;
				return NULL;
			}
			once_list = expanded_list;
		}
		once_list[once_list_len - 1] = strdup(file);
		
		return NULL;
	} else {
		// Skip if the file is in the "include once" list
		for (size_t i = 0; i < once_list_len; ++i) {
			if (strcmp(once_list[i], file) == 0) return NULL;
		}
	}
	
	// Open the source file
	char *code;
	FILE *source_file = fopen(file, "r");
	if (source_file) {
		// Read the source file
		code = readfile(source_file);
		fclose(source_file);
		if (!code) die("Failed to read from source file!");
	} else {
		code = malloc(1);
		code[0] = '\0';
	}
	
	// The following is required because the parser needs a string with two null terminators
	*size = strlen(code);
	char *code_padded = realloc(code, *size + 2);
	if (!code_padded) {
		free(code);
		die("Failed to expand code buffer");
	}
	code_padded[*size + 1] = '\0';
	
	return code_padded;
}

int main(int argc, char *argv[]) {
	if (argc < 2) die("No arguments!");
	
	// Parse the code
	parse(argv[1], provide_code);
	
	return EXIT_SUCCESS;
}

It was surprisingly easy to write a fully featured scanner/tokenizer in flex, guess I should have used it from the beginning. Now I can start work on the parser, hopefully it will be easy too, surely easier than writing it all manually 🥵

seadoggie

TheDcoder /* TODO: #include-once */

Hehe

Also, I like that the online C formatter can't handle escaped strings

TheDcoder

I forgot to remove that comment.

seadoggie the online C formatter can't handle escaped strings

To be fair, the code is a mix of RegEx and C, and we all know how crazy regex can get 😛

TheDcoder

I think this is a new record, another update just before a week passes! 🥳

I am here to present the first prototype of the new bison parser:

// parser.y
/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2021 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */

%code top {
	#define _GNU_SOURCE /* Required to enable (v)asprintf */
	#include <stdio.h>
	#include <string.h>
}

%union {
	char *str;
}

%token UNKNOWN
%token WS
%token COMMENT DIRECTIVE
%token <str> NUMBER STRING BOOL
%token WORD
%token MACRO VARIABLE
%token OPERATOR BRACKET DOT COMMA

 /* Operators */
%precedence '?'
%precedence ':'
%left AND "And" OR "Or"
%left LT '<' GT '>' LTE "<=" GTE ">=" EQU '=' NEQ "<>" SEQU "=="
%left '&'
%left '+' '-'
%left '*' '/'
%left '^'
%left NOT "Not"
%precedence INVERSION
%precedence GROUPING

%type <str> expression

%{
int yylex();
void yyerror(const char *s);
%}

%%

exp-list: /* nothing */
	| expression {puts($1);}

expression: NUMBER {$$ = strdup($1);}
	| expression '?' expression ':' expression {asprintf(&$$, " (%s ? %s : %s) ", $1, $3, $5);}
	| expression "And" expression {asprintf(&$$, " (%s And %s) ", $1, $3);}
	| expression "Or" expression {asprintf(&$$, " (%s Or %s) ", $1, $3);}
	| expression '<' expression {asprintf(&$$, " (%s < %s) ", $1, $3);}
	| expression '>' expression {asprintf(&$$, " (%s > %s) ", $1, $3);}
	| expression '=' expression {asprintf(&$$, " (%s = %s) ", $1, $3);}
	| expression "<=" expression {asprintf(&$$, " (%s <= %s) ", $1, $3);}
	| expression ">=" expression {asprintf(&$$, " (%s >= %s) ", $1, $3);}
	| expression "<>" expression {asprintf(&$$, " (%s <> %s) ", $1, $3);}
	| expression "==" expression {asprintf(&$$, " (%s == %s) ", $1, $3);}
	| expression '&' expression {asprintf(&$$, " (%s & %s) ", $1, $3);}
	| expression '+' expression {asprintf(&$$, " (%s + %s) ", $1, $3);}
	| expression '-' expression {asprintf(&$$, " (%s - %s) ", $1, $3);}
	| expression '*' expression {asprintf(&$$, " (%s * %s) ", $1, $3);}
	| expression '/' expression {asprintf(&$$, " (%s / %s) ", $1, $3);}
	| expression '^' expression {asprintf(&$$, " (%s ^ %s) ", $1, $3);}
	| "Not" expression {asprintf(&$$, " (Not %s) ", $2);}
	| '-' expression %prec INVERSION {asprintf(&$$, " (-%s) ", $2);}
	| '(' expression ')' %prec GROUPING {asprintf(&$$, " (%s) ", $2);}

%%

void start_parser() {
	yyparse();
}

void yyerror(char const *s) {
	fputs(s, stderr);
	fputs("\n", stderr);
}

This version can only handle numeric expressions, and the output is just a string with an excessive amount of brackets to show the correct order of evaluation. It ain't pretty but it does the job 😛

Here are a few sample inputs I tried:

> echo '1 + 2' | ./eci /dev/fd/0
 (1 + 2) 
> echo '-((1 + 2) / 3) * 4' | ./eci /dev/fd/0
 ( (- ( ( ( (1 + 2) )  / 3) ) )  * 4) 
> echo '42 ? 123 ? 86 : -((1 + 2) / 3) * 4 : 007' | ./eci /dev/fd/0
 (42 ?  (123 ? 86 :  ( (- ( ( ( (1 + 2) )  / 3) ) )  * 4) )  : 007)

(the output is the line of text below the > shell prompt)

Even though this is very primitive, it has served as proof that I can use bison to write a proper parser, now I can move onto the more intricate details of parsing such as evaluating values and constructing a syntax tree to represent the code.

I will keep you guys updated on the progress, hope the next update comes as soon as this one 🤞

argumentum

if au3check does check the source, maybe the coder of it can lend a hand, or give an educated opinion with a chunk of code ?.
...just an idea. 🙂

TheDcoder

Au3Check was written by Tylo [1] and he hasn't visited the AutoIt forum in a decade, so I doubt we can get his opinion 😅

Jos has however offered to publicly release the code [1] if Jon is okay with it, but we haven't received any response, so I guess we aren't going to get our hands on that code.

It shouldn't really matter though, it is easy enough to write parser code in bison and I doubt I'd verbatim use the old code anyway, I like to do my own versions 😀

https://www.autoitscript.com/forum/topic/198464-a-cross-platform-implementation-of-the-autoit-language/?do=findComment&comment=1436876

argumentum

It was just an idea. There is also the v3.1 code and whatever version of AHK in the net, so, plenty of refences of working code.
I don't know why I thought exclusively of that at the moment. Brain fart 😅

TheDcoder

argumentum AutoIt 3 has a handmade parser written by Jon, much like how I was planning to do myself until I realized the effort was too much for an initial version.

As for AHK, their syntax is very different, I think it is similar to AutoIt 2 (which I never used).

Don't worry about a reference, I have plenty in the books and on the internet 🙂

TheDcoder

I apologize for the lack of updates, life threw a bunch of stuff at me again so I couldn't focus on the project very much, however I think I can now resume my focus as I have mostly dealt with the other stuff 🙂

Behind the scenes I have been working on two helper libraries (UDFs in AutoIt-speak):

alloc - This is a wrapper around dynamic memory allocation functions (like malloc), its primary purpose is to keep record of all allocated memory for a certain context.
cease - This is a wrapper around setjmp and longjmp, it is similar to the raise_error functions we had in the old parser.

Both of these libraries assist in error recovery, alloc makes it easier to clean up all the memory and cease will make it easier to return to a point in the program where we can safely clean up everything and resume routine operation.

With these two now in place, I can resume work on the parser!

Here is the current code I have if anyone is interested:

Click to reveal Click to hide

alloc.h:

/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2021 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */

#ifndef ALLOC_H
#define ALLOC_H

#include <stddef.h>
#include "cease/cease.h"

typedef void *AllocatorFunc(size_t);
typedef void AllocatorFreeFunc(void *);

struct AllocatorNode {
	// FIFO for better performance
	void *ptr;
	struct AllocatorNode *prev;
};

struct Allocator {
	struct AllocatorNode *node;
	AllocatorFunc *alloc;
	AllocatorFreeFunc *free;
	CeasePoint *point;
	char *ctx;
};
typedef struct Allocator Allocator;

Allocator alloc_init(AllocatorFunc *alloc, AllocatorFreeFunc *free, CeasePoint *cease_point, char *def_ctx);
void *alloc_new(Allocator *allocator, size_t size);
void *alloc(Allocator *allocator, size_t size, char *ctx);
void alloc_free(Allocator *allocator, void *ptr);
void alloc_free_all(Allocator *allocator);

#endif

alloc.c:

/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2021 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */

#include <stdbool.h>
#include "alloc.h"
#include "cease/cease.h"

Allocator alloc_init(AllocatorFunc *alloc, AllocatorFreeFunc *free, CeasePoint *cease_point, char *def_ctx) {
	return (Allocator){
		.alloc = alloc,
		.free = free,
		.point = cease_point,
		.ctx = def_ctx,
		.node = NULL
	};
}

void *alloc_new(Allocator *allocator, size_t size) {
	AllocatorFunc *aalloc = allocator->alloc;
	AllocatorFreeFunc *afree = allocator->free;
	struct AllocatorNode *node = aalloc(sizeof *node);
	if (!node) goto nomem;
	node->ptr = aalloc(size);
	if (!node->ptr) {
		afree(node);
		goto nomem;
	}
	node->prev = allocator->node;
	allocator->node = node;
	return node->ptr;
	
	nomem: // alloc function returned NULL
	if (allocator->point) cease_mem(allocator->point, allocator->ctx);
	return NULL;
}

void *alloc(Allocator *allocator, size_t size, char *ctx) {
	if (!ctx) return alloc_new(allocator, size);
	char *def_ctx = allocator->ctx;
	allocator->ctx = ctx;
	void *ptr = alloc_new(allocator, size);
	allocator->ctx = def_ctx;
	return ptr;
}

void alloc_free(Allocator *allocator, void *ptr) {
	AllocatorFreeFunc *afree = allocator->free;
	struct AllocatorNode *node = allocator->node;
	struct AllocatorNode *next_node = NULL;
	while (node) {
		if (node->ptr != ptr) {
			next_node = node;
			node = node->prev;
			continue;
		}
		if (next_node) {
			next_node->prev = node->prev;
		} else {
			// node == allocator->node
			allocator->node = node->prev;
		}
		afree(ptr);
		afree(node);
		break;
	}
}

void alloc_free_all(Allocator *allocator) {
	AllocatorFreeFunc *afree = allocator->free;
	struct AllocatorNode *node = allocator->node;
	struct AllocatorNode *prev_node;
	while (node) {
		prev_node = node->prev;
		afree(node->ptr);
		afree(node);
		node = prev_node;
	}
	allocator->node = NULL;
}

cease.h:

/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2021 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */

#ifndef CEASE_H
#define CEASE_H

#include <setjmp.h>
#include <stdbool.h>
#include <stdnoreturn.h>

struct CeasePoint {
	jmp_buf jump;
	char *msg;
	bool free_msg;
};

typedef struct CeasePoint CeasePoint;

CeasePoint cease_get_point(void);
noreturn void cease(CeasePoint *point, char *msg, bool free_msg);
noreturn void cease_fmt(CeasePoint *point, char *def, char *fmt, ...);
noreturn void cease_mem(CeasePoint *point, char *context);

#endif

cease.c:

/* 
 * This file is part of EasyCodeIt.
 * 
 * Copyright (C) 2021 TheDcoder <TheDcoder@protonmail.com>
 * 
 * EasyCodeIt is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */

#define _GNU_SOURCE /* Required to enable (v)asprintf */
#include <setjmp.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdnoreturn.h>
#include "cease.h"

CeasePoint cease_get_point(void) {
	return (CeasePoint){.msg = NULL};
}

noreturn void cease(CeasePoint *point, char *msg, bool free_msg) {
	if (point->free_msg && point->msg) free(point->msg);
	
	point->msg = msg;
	point->free_msg = free_msg;
	
	longjmp(point->jump, true);
}

noreturn void cease_fmt(CeasePoint *point, char *def, char *fmt, ...) {
	va_list args;
	va_start(args, fmt);
	
	char *msg;
	int result = vasprintf(&msg, fmt, args);
	if (result == -1) msg = def;
	
	va_end(args);
	
	cease(point, msg, result != -1);
}

noreturn void cease_mem(CeasePoint *point, char *context) {
	static char *msg = "Failed to allocate memory";
	if (context) {
		cease_fmt(point, msg, "%s when %s", msg, context);
	} else {
		cease(point, msg, false);
	};
}

TheDcoder

New Update: https://forum.dtw.tools/d/29-easycodeit-expression-parser-download-available

TheDcoder

Hello everyone,

I know that it's been a while since the last update... 1 year 4 months 21 days to be exact. A passionate member of the AutoIt forum Chris ("JockoDundee") also passed away shortly after the previous update. So this update is dedicated to him 🙂

A lot of stuff happened in-between and this project essentially went into a coma given my lack of motivation to keep it going forward. I myself was going through a depressive period where I was very unproductive, I even stopped working on work-related projects, only working on existing client contracts occasionally to keep the boat afloat essentially, relying on savings to cover the gap.

Anyway, keeping all of that aside. Today I finally convinced myself to get back to what I was working on behind the curtain.

I had already begun work on parsing statements, but hit a road-block while trying to create a list of expressions separated by new lines (statements, essentially).

Today I got the idea to ask ChatGPT, the AI, for help after reading countless articles and posts on it. Apparently it's even on track to replace programmers entirely... at-least that's what some people say, I have my reservations 😆

I have to admit though, while not a complete replacement, it is a very competent wing-man and it actually helped me fix a bug which I had banging my head around trying to debug! Of course it wasn't perfect, but you can easily guide it to the right direction if you know what you are doing.

Here's what the issue I had to deal with, I created a new expression_list rule to match a list of expressions:

expression_list:
	  expression {$$ = exprlist_from_expr(&$1, NULL);}
	| expression '\n' expression_list {$$ = exprlist_from_expr(&$1, &$3);} */
	| expression_list ',' expression {$$ = exprlist_from_expr(&$3, &$1);}
	| expression_list '\n' expression {$$ = exprlist_from_expr(&$3, &$1);}

But it kept throwing a syntax error after parsing the first expression... and I could never figure out why. ChatGPT came to my rescue, and after making it familiar with the code and issue, it suggested the following approach:

expression_list:
	  expression {$$ = exprlist_from_expr(&$1, NULL);}
	| expression '\n' expression_list {$$ = exprlist_from_expr(&$1, &$3);} */
	| expression_list ',' expression {$$ = exprlist_from_expr(&$3, &$1);}
	| expression_list NL expression {$$ = exprlist_from_expr(&$3, &$1);}
	| expression_list NL {$$ = $1;} // Allow multiple new lines between expressions

And this actually works like a charm! The change is simple, the literal '\n' token is replaced with the NL token from the lexer.

To be honest I still don't know why this worked and not the older code, but I'm glad to be past this road-block. Maybe it's because the lexer ate up the \n character so bison couldn't see it?

I plan to continue work on this so we have a working parser, and I can finally begin work on the interpreter and runtime 😁