PHP 5.4.31 Released

token_get_all

(PHP 4 >= 4.2.0, PHP 5)

token_get_allSpaltet angegebenen PHP-Quelltext in PHP-Tokens auf

Beschreibung

array token_get_all ( string $source )

token_get_all() parst den angegebenen Quelltext source und spaltet ihn mit Hilfe des lexikalischen Scanners der Zend-Engine in PHP-Sprachtokens auf.

Eine Übersicht über die verschiedenen Tokens finden Sie im Liste der Parsertoken oder Sie können die Funktion token_name() zum Übersetzen eines Tokens in seinen Textrepäsentanten verwenden.

Parameter-Liste

source

PHP-Quelltext der geparst werden soll.

Rückgabewerte

Array mit Tokenbezeichner. Jeder einzelne Bezeichner ist entweder ein einzelnes Zeichen (z.B.: ;, ., >, !, usw.), oder ein Array bestehend aus drei Elementen. Das erste Element enthält den Tokenindex, das zweite Element den originalen Textinhalt und die Zeilennummer im dritten Element.

Beispiele

Beispiel #1 token_get_all() Beispiele

<?php
$tokens 
token_get_all('<?php echo; ?>'); /* => array(
                                                  array(T_OPEN_TAG, '<?php'),
                                                  array(T_ECHO, 'echo'),
                                                  ';',
                                                  array(T_CLOSE_TAG, '?>') ); */

/* Beachten Sie, dass im folgenden Beispiel die Zeichenkette geparst wurde
   als T_INLINE_HTML und nicht wie erwartet als T_COMMENT (T_ML_COMMENT in PHP <5).
   Das liegt an den fehlenden Beginn-/Endetags die im "Quelltext" verwendet wurden.
   Das entspricht einem Kommentar außerhalb der <?php ?> Tags in einer normalen Datei. */
$tokens token_get_all('/* comment */'); // => array(array(T_INLINE_HTML, '/* comment */'));
?>

Changelog

Version Beschreibung
5.2.2 Die Zeilennummer wird im dritten Element zurückgegeben

add a note add a note

User Contributed Notes 5 notes

up
3
gomodo at free dot fr
4 years ago
Yes, some problems (On WAMP, PHP 5.3.0 ) with get_token_all()

1 : bug line numbers
Since PHP 5.2.2 token_get_all()  should return Line numbers in element 2..
.. but for instance (5.3.0 on WAMP), it work perfectly only with PHP code (not HMTL miwed), but if you have some T_INLINE_HTML detected by token_get_all() ,  sometimes you find wrongs line numbers  (return next line)... :(

2: bug warning message can impact loops
Warning with php code uncompleted (ex : php code line by line) :
for example if a comment tag is not closed  token_get_all()  can block loops on this  warning :
Warning: Unterminated comment starting line

This problem seem not occur in CLI mod (php command line), but only in web mod.

Waiting more stability, used token_get_all()  only on PHP code (not HMTL miwed) :
First extract entirely PHP code (with open et close php tag),
Second use token_get_all()  on the pure PHP code.

3 : Why there not function to extract PHP code (to extract HTML, we have Tidy..)?

Waiting, I used a function :

The code at end this post :
http://www.developpez.net/forums/d786381/php/langage/
fonctions/analyser-fichier-php-token_get_all/

This function not support :
- Old notation :  "<?  ?>" and "<% %>"
- heredoc syntax
- nowdoc syntax (since PHP 5.3.0)
up
1
rogier
6 years ago
Complementary note to code below:
Note that only the FIRST 2 (or 3, if needed) array elements will be updated.

Since I only encountered incorrect results on the FIRST occurence of T_OPEN_TAG, I wrote this quick fix.
Any other following T_OPEN_TAG are, on my testing system (Apache 2.0.52, PHP 5.0.3), parsed correctly.

So, This function assumes only a possibly incorrect first T_OPEN_TAG.
Also, this function assumes the very first element (and ONLY the first element) of the token array to be the possibly incorrect token.
This effectively translates to the first character of the tokenized source to be the start of a php script opening tag '<', followed by either 'php' OR '%' (ASP_style)
up
1
Dennis Robinson from basnetworks dot net
5 years ago
I wanted to use the tokenizer functions to count source lines of code, including counting comments.  Attempting to do this with regular expressions does not work well because of situations where /* appears in a string, or other situations.  The token_get_all() function makes this task easy by detecting all the comments properly.  However, it does not tokenize newline characters.  I wrote the below set of functions to also tokenize newline characters as T_NEW_LINE.

<?php

define
('T_NEW_LINE', -1);

function
token_get_all_nl($source)
{
   
$new_tokens = array();

   
// Get the tokens
   
$tokens = token_get_all($source);

   
// Split newlines into their own tokens
   
foreach ($tokens as $token)
    {
       
$token_name = is_array($token) ? $token[0] : null;
       
$token_data = is_array($token) ? $token[1] : $token;

       
// Do not split encapsed strings or multiline comments
       
if ($token_name == T_CONSTANT_ENCAPSED_STRING || substr($token_data, 0, 2) == '/*')
        {
           
$new_tokens[] = array($token_name, $token_data);
            continue;
        }

       
// Split the data up by newlines
       
$split_data = preg_split('#(\r\n|\n)#', $token_data, -1, PREG_SPLIT_DELIM_CAPTURE | PREG_SPLIT_NO_EMPTY);

        foreach (
$split_data as $data)
        {
            if (
$data == "\r\n" || $data == "\n")
            {
               
// This is a new line token
               
$new_tokens[] = array(T_NEW_LINE, $data);
            }
            else
            {
               
// Add the token under the original token name
               
$new_tokens[] = is_array($token) ? array($token_name, $data) : $data;
            }
        }
    }

    return
$new_tokens;
}

function
token_name_nl($token)
{
    if (
$token === T_NEW_LINE)
    {
        return
'T_NEW_LINE';
    }

    return
token_name($token);
}

?>

Example usage:

<?php

$tokens
= token_get_all_nl(file_get_contents('somecode.php'));

foreach (
$tokens as $token)
{
    if (
is_array($token))
    {
        echo (
token_name_nl($token[0]) . ': "' . $token[1] . '"<br />');
    }
    else
    {
        echo (
'"' . $token . '"<br />');
    }
}

?>

I'm sure you can figure out how to count the lines of code, and lines of comments with these functions.  This was a huge improvement on my previous attempt at counting lines of code with regular expressions.  I hope this helps someone, as many of the user contributed examples on this website have helped me in the past.
up
0
kevin at metalaxe dot com
6 years ago
Rogier, thanks for that fix. This bug still exists in php 5.2.5. I did notice though that it is possible for a notice to pop up from your code. Changing this line:

            $temp[] = $tokens[0][2];

To read this:

            $temp[] = isset($tokens[0][2])?$tokens[0][2]:'unknown';

fixes this notice.
To Top