tokenizer Struct Reference

The tokenizer struct. More...

#include <tokenizer.h>

Data Fields
wchar_t *	buff
	A pointer into the original string, showing where the next token begins.
wchar_t *	orig_buff
	A copy of the original string.
wchar_t *	last
	A pointer to the last token.
int	last_type
	Type of last token.
int	last_len
	Length of last token.
int	last_pos
	Offset of last token.
int	has_next
	Whether there are more tokens.
int	accept_unfinished
	Whether incomplete tokens are accepted.
int	show_comments
	Whether commants should be returned.
int	free_orig
	Flag set to true of the orig_buff points to an internal string that needs to be free()d when deallocating the tokenizer.
wchar_t	last_quote
	Type of last quote, can be either ' or ".
int	error
	Last error.

---

Detailed Description

The tokenizer struct.

---

Field Documentation

int tokenizer::free_orig

Flag set to true of the orig_buff points to an internal string that needs to be free()d when deallocating the tokenizer.

Referenced by tok_destroy().

wchar_t tokenizer::last_quote

Type of last quote, can be either ' or ".

Referenced by read_string(), and tok_last_quote().

---

The documentation for this struct was generated from the following file:

• tokenizer.h