about summary refs log tree commit diff
path: root/src/libexpr/lexer.l
AgeCommit message (Collapse)AuthorFilesLines
2018-10-27 simplify handling of extra '}'Guillaume Maudoux1-18/+15
2018-08-29 libexpr: Use int64_t for NixIntaszlig1-2/+6
Using a 64bit integer on 32bit systems will come with a bit of a performance overhead, but given that Nix doesn't use a lot of integers compared to other types, I think the overhead is negligible also considering that 32bit systems are in decline. The biggest advantage however is that when we use a consistent integer size across all platforms it's less likely that we miss things that we break due to that. One example would be: https://github.com/NixOS/nixpkgs/pull/44233 On Hydra it will evaluate, because the evaluator runs on a 64bit machine, but when evaluating the same on a 32bit machine it will fail, so using 64bit integers should make that consistent. While the change of the type in value.hh is rather easy to do, we have a few more options available for doing the conversion in the lexer: * Via an #ifdef on the architecture and using strtol() or strtoll() accordingly depending on which architecture we are. For the #ifdef we would need another AX_COMPILE_CHECK_SIZEOF in configure.ac. * Using istringstream, which would involve copying the value. * As we're already using boost, lexical_cast might be a good idea. Spoiler: I went for the latter, first of all because lexical_cast does have an overload for const char* and second of all, because it doesn't involve copying around the input string. Also, because istringstream seems to come with a bigger overhead than boost::lexical_cast: https://www.boost.org/doc/libs/release/doc/html/boost_lexical_cast/performance.html The first method (still using strtol/strtoll) also wasn't something I pursued further, because it is also locale-aware which I doubt is what we want, given that the regex for int is [0-9]+. Signed-off-by: aszlig <aszlig@nix.build> Fixes: #2339
2018-05-11 Don't return negative numbers from the flex tokenizerEelco Dolstra1-1/+5
Fixes #1374. Closes #2129.
2018-05-11 Revert "Throw a specific error for incomplete parse errors."Eelco Dolstra1-2/+0
This reverts commit 6498adb002bcf7e715afe46c23b8635d4592c156. We don't actually use IncompleteParseError in 'nix repl'.
2018-03-02 libexpr: Recognize newline in more places in lexerTuomas Tynkkynen1-4/+5
Flex's regexes have an annoying feature: the dot matches everything except a newline. This causes problems for expressions like: "${0}\ " where the backslash-newline combination matches this rule instead of the intended one mentioned in the comment: <STRING>\$|\\|\$\\ { /* This can only occur when we reach EOF, otherwise the above (...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered. This is technically invalid, but we leave the problem to the parser who fails with exact location. */ return STR; } However, the parser actually accepts the resulting token sequence ('"' DOLLAR_CURLY 0 '}' STR '"'), which is a problem because the lexer rule didn't assign anything to yylval. Ultimately this leads to a crash when dereferencing a NULL pointer in ExprConcatStrings::bindVars(). The fix does change the syntax of the language in some corner cases but I think it's only turning previously invalid (or crashing) syntax to valid syntax. E.g. "a\ b" and ''a''\ b'' were previously syntax errors but now both result in "a\nb". Found by afl-fuzz.
2018-02-16 libexpr: Pre-reserve space in string in unescapeStr()Tuomas Tynkkynen1-3/+4
Avoids some malloc() traffic.
2017-11-14 Revert "Don't parse "x:x" as a URI"Eelco Dolstra1-1/+1
This reverts commit f90f660b243866b8860eeb24cc4a345d32cc7ce7. This broke Hydra's release.nix, which contained preCheck = ''export LOGNAME=${LOGNAME:-foo}'';
2017-10-30 Don't parse "x:x" as a URIEelco Dolstra1-1/+1
URIs now have to contain "://" or start with "channel:".
2017-07-30 Replace Unicode quotes in user-facing strings by ASCIIJörg Thalheim1-4/+4
Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-05-01 lexer: remove catch-all rules hiding real errorsGuillaume Maudoux1-10/+17
With catch-all rules, we hide potential errors. It turns out that a4744254 made one cath-all useless. Flex detected that is was impossible to reach. The other is more subtle, as it can only trigger on unfinished escapes in unfinished strings, which only occurs at EOF.
2017-05-01 Fix lexer to support `$'` in multiline strings.Guillaume Maudoux1-1/+2
2016-12-06 Tweak error messageEelco Dolstra1-2/+2
2016-11-27 Improve error message on trailing path slashesGuillaume Maudoux1-4/+12
2016-11-13 Fix comments parsingGuillaume Maudoux1-1/+1
Fixed the parsing of multiline strings ending with an even number of stars, like /** this **/. Added test cases for comments.
2016-02-24 Throw a specific error for incomplete parse errors.Scott Olson1-0/+2
`nix-repl` will use this for deciding whether to keep waiting for input or error out right away.
2016-02-12 Merge pull request #762 from ctheune/ctheune-floatsEelco Dolstra1-0/+7
Implement floats
2016-01-20 Revert "Revert "next try for "don't abort when given unmatched '}' with ↵Eelco Dolstra1-7/+17
'start-condition stack underflow'. This fixes #751""" This reverts commit b669d3d2e83d3c50238751b57cff3ed0ca39bc8a.
2016-01-20 Revert "next try for "don't abort when given unmatched '}' with ↵Eelco Dolstra1-17/+7
'start-condition stack underflow'. This fixes #751"" This reverts commit ed23c8568e10d15196bb4ff2b79fc14191d28109. Let's merge this *after* the 1.11.1 release.
2016-01-19 next try for "don't abort when given unmatched '}' with 'start-condition ↵Fabian Schmitthenner1-7/+17
stack underflow'. This fixes #751" This reverts commit 8120b6fb8a4924f8ae717bba9bbda4a2f89e2141 and fixes the regression introduced in 8d22b26448a091c76ab972c0b0603daac5e255e4.
2016-01-19 Revert "don't abort when given unmatched '}' with 'start-condition stack ↵Eelco Dolstra1-17/+7
underflow'. This fixes #751" This reverts commit 8d22b26448a091c76ab972c0b0603daac5e255e4. It breaks Nixpkgs: $ nix-env -qa error: syntax error, unexpected IND_STR, expecting '}', at /home/eelco/Dev/nixpkgs-stable/pkgs/top-level/python-packages.nix:7605:8
2016-01-12 don't abort when given unmatched '}' with 'start-condition stack underflow'. ↵Fabian Schmitthenner1-7/+17
This fixes #751
2016-01-05 Edge condition: parser did not pick up floats starting exactly with 0.Christian Theune1-1/+1
2016-01-05 Fix up float parsing.Christian Theune1-2/+2
2016-01-05 Try a simplified version of float lexing that didn't work.Christian Theune1-1/+1
The last one I tried was botchered anyway ...
2016-01-05 First hit at providing support for floats in the language.Christian Theune1-0/+7
2015-07-03 Fix the parsing of "$"'s in strings.Guillaume Maudoux1-6/+5
2015-07-03 Fix the hack that resets the scanner state.Guillaume Maudoux1-27/+15
2015-02-19 Allow the leading component of a path to be a ~Shea Levy1-0/+2
2014-08-20 Use proper quotes everywhereEelco Dolstra1-1/+1
2014-01-14 Allow "bare" dynamic attrsShea Levy1-0/+2
Now, in addition to a."${b}".c, you can write a.${b}.c (applicable wherever dynamic attributes are valid). Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-02 Fix whitespaceEelco Dolstra1-5/+5
2013-08-19 Store Nix integers as longsEelco Dolstra1-2/+4
So on 64-bit systems, integers are now 64-bit. Fixes #158.
2013-08-02 Add comparison operators ‘<’, ‘<=’, ‘>’ and ‘>=’Eelco Dolstra1-0/+2
2013-03-14 Fix building against Bison 2.6Eelco Dolstra1-1/+0
2012-09-27 Allow dashes in identifiersEelco Dolstra1-1/+1
In Nixpkgs, the attribute in all-packages.nix corresponding to a package is usually equal to the package name. However, this doesn't work if the package contains a dash, which is fairly common. The convention is to replace the dash with an underscore (e.g. "dbus-lib" becomes "dbus_glib"), but that's annoying. So now dashes are valid in variable / attribute names, allowing you to write: dbus-glib = callPackage ../development/libraries/dbus-glib { }; and buildInputs = [ dbus-glib ]; Since we don't have a negation or subtraction operation in Nix, this is unambiguous.
2011-08-06 * Add a Nix expression search path feature. Paths between angleEelco Dolstra1-0/+2
brackets, e.g. import <nixpkgs/pkgs/lib> are resolved by looking them up relative to the elements listed in the search path. This allows us to get rid of hacks like import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib" The search path can be specified through the ‘-I’ command-line flag and through the colon-separated ‘NIX_PATH’ environment variable, e.g., $ nix-build -I /etc/nixos ... If a file is not found in the search path, an error message is lazily thrown.
2011-07-13 * Allow a default value in attribute selection by writingEelco Dolstra1-0/+1
x.y.z or default (as originally proposed in https://mail.cs.uu.nl/pipermail/nix-dev/2009-September/002989.html). For instance, an expression like stdenv.lib.attrByPath ["features" "ckSched"] false args can now be written as args.features.ckSched or false
2010-10-23 * Optimise string constants by putting them in the symbol table.Eelco Dolstra1-4/+4
2010-05-06 * Store attribute positions in the AST and report duplicate attributeEelco Dolstra1-5/+8
errors with position info. * For all positions, use the position of the first character of the first token, rather than the last character of the first token plus one.
2010-04-21 * Fix the interpretation of ''\<character> in indented strings.Eelco Dolstra1-1/+1
2010-04-12 * Indented strings.Eelco Dolstra1-9/+10
2010-04-12 * More missing constructs.Eelco Dolstra1-3/+1
2010-04-12 * Don't use ATerms for the abstract syntax trees anymore. NotEelco Dolstra1-12/+12
finished yet.
2010-04-12 * Don't use ATerms to represent integers in the lexer.Eelco Dolstra1-1/+1
2008-08-14 * Added an experimental feature suggested by Andres: ellipses ("...")Eelco Dolstra1-0/+1
in attribute set pattern matches. This allows defining a function that takes *at least* the listed attributes, while ignoring additional attributes. For instance, {stdenv, fetchurl, fuse, ...}: stdenv.mkDerivation { ... }; defines a function that requires an attribute set that contains the specified attributes but ignores others. The main advantage is that we can then write in all-packages.nix aefs = import ../bla/aefs pkgs; instead of aefs = import ../bla/aefs { inherit stdenv fetchurl fuse; }; This saves a lot of typing (not to mention not having to update all-packages.nix with purely mechanical changes). It saves as much typing as the "args: with args;" style, but has the advantage that the function arguments are properly declared (not implicit in what the body of the "with" uses).
2008-02-05 * Fix the parsing ofEelco Dolstra1-1/+5
'' '${foo}' '' where the antiquote should work as expected, instead of giving the string "'${foo}'".
2007-12-06 * Syntax to escape '', ${.Eelco Dolstra1-0/+12
2007-11-30 * Added a new kind of multi-line string literal delimited by twoEelco Dolstra1-0/+15
single quotes. Example (from NixOS): job = '' start on network-interfaces start script rm -f /var/run/opengl-driver ${if videoDriver == "nvidia" then "ln -sf ${nvidiaDrivers} /var/run/opengl-driver" else if cfg.driSupport then "ln -sf ${mesa} /var/run/opengl-driver" else "" } rm -f /var/log/slim.log end script ''; This style has two big advantages: - \, ' and " aren't special, only '' and ${. So you get a lot less escaping in shell scripts / configuration files in Nixpkgs/NixOS. The delimiter '' is rare in scripts (and can usually be written as ""). ${ is also fairly rare. Other delimiters such as <<...>>, {{...}} and <|...|> were also considered but this one appears to have the fewest drawbacks (thanks Martin). - Indentation is intelligently stripped so that multi-line strings can follow the nesting structure of the containing Nix expression. E.g. in the example above 6 spaces are stripped from the start of each line. This prevents unnecessary indentation in generated files (which sometimes even breaks things). See tests/lang/eval-okay-ind-string.nix for some examples.
2006-10-16 * Big cleanup of the semantics of paths, strings, contexts, stringEelco Dolstra1-1/+1
concatenation and string coercion. This was a big mess (see e.g. NIX-67). Contexts are now folded into strings, so that they don't cause evaluation errors when they're not expected. The semantics of paths has been clarified (see nixexpr-ast.def). toString() and coerceToString() have been merged. Semantic change: paths are now copied to the store when they're in a concatenation (and in most other situations - that's the formalisation of the meaning of a path). So "foo " + ./bla evaluates to "foo /nix/store/hash...-bla", not "foo /path/to/current-dir/bla". This prevents accidental impurities, and is more consistent with the treatment of derivation outputs, e.g., `"foo " + bla' where `bla' is a derivation. (Here `bla' would be replaced by the output path of `bla'.)
2006-10-02 * Finally, a real "let" syntax: `let x = ...; ... z = ...; in ...'.Eelco Dolstra1-0/+1
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-08-31T01·00+0000