From d134b4220c9eddf370afd2763c07555731b7f088 Mon Sep 17 00:00:00 2001 From: Peter Norvig Date: Tue, 13 Jan 2026 21:51:05 -0800 Subject: [PATCH] Add files via upload --- ipynb/Goldberg.ipynb | 2141 +++++++++++++++++------------------------- 1 file changed, 850 insertions(+), 1291 deletions(-) diff --git a/ipynb/Goldberg.ipynb b/ipynb/Goldberg.ipynb index ce374e7..f69d8a1 100644 --- a/ipynb/Goldberg.ipynb +++ b/ipynb/Goldberg.ipynb @@ -18,42 +18,27 @@ "## Definitions\n", "\n", "- A **generative language model** is a model that, when given an initial text, can predict what tokens come next; it can generate a continuation of a partial text. (And when the initial text is empty, it can generate the whole text.) In terms of probabilities, the model represents *P*(*t* | *h*), the probability distribution that the next token will be *t*, given a history of previous tokens *h*. The probability distribution is estimated by looking at a training corpus of text.\n", - "\n", - "- A **token** is a unit of text. It can be a single character (as covered by Karpathy and Goldberg) or more generally it can be a word or a part of a word (as allowed in my implementation).\n", - "\n", + "- A **token** is a unit of text. In a character model, \"walking\" would be 7 tokens, one for each letter, while in a word model it would be one token, and in other models it might be two tokens (\"walk\", \"ing\").\n", "- A generative model stands in contrast to a **discriminative model**, such as an email spam filter, which can discriminate between spam and non-spam, but can't be used to generate a new sample of spam.\n", - "\n", - "\n", - "- An **n-gram model** is a generative model that estimates the probability of *n*-token sequences. For example, a 5-gram character model would be able to say that given the previous 4 characters `'chai'`, the next character might be `'r'` or `'n'` (to form `'chair'` or `'chain'`). A 5-gram model is also called a model of **order** 4, because it maps from the 4 previous tokens to the next token.\n", - "\n", - "- A **recurrent neural network (RNN) model** is more powerful than an *n*-gram model, because it contains memory units that allow it to retain information from more than *n* tokens. See Karpathy for [details](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).\n", - "\n", - "- Current **large language models** such as ChatGPT, Claude, Llama, and Gemini use an even more powerful model called a [transformer](https://en.wikipedia.org/wiki/Transformer_%28deep_learning_architecture%29). Karpathy has [an introduction](https://www.youtube.com/watch?v=zjkBMFhNj_g&t=159s).\n", + "- An **n-gram model** is a generative model that estimates the probability of *n*-token sequences. For example, a 5-gram character model would be able to say that given the previous 4 characters `'chai'`, the next character might be `'r'` or `'n'` (to form `'chair'` or `'chain'`). A 5-gram model is also called a [Markov model](https://en.wikipedia.org/wiki/Markov_model) of **order** 4, because it maps from the 4 previous tokens to the next token.\n", + "- A **recurrent neural network (RNN) model** is more powerful than an *n*-gram model, because it contains memory units that allow it to retain some information from more than *n* tokens in the past. See Karpathy for [details](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).\n", + "- Current **large language models** such as ChatGPT, Claude, and Gemini use a more powerful model called a [transformer](https://en.wikipedia.org/wiki/Transformer_%28deep_learning_architecture%29). Karpathy has [an introduction](https://www.youtube.com/watch?v=zjkBMFhNj_g&t=159s).\n", "\n", "## Training Data\n", "\n", - "A language model learns probabilities by observing a corpus of text that we call the **training data**. \n", + "A language model learns probabilities by counting token subsequences in a corpus of text that we call the **training data**. \n", "\n", - "Both Karpathy and Goldberg use the works of Shakespeare (about 800,000 words) as their initial training data:" + "Both Karpathy and Goldberg use the works of Shakespeare as their initial training data:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " 167204 832301 4573338 shakespeare_input.txt\n" - ] - } - ], + "outputs": [], "source": [ - "! [ -f shakespeare_input.txt ] || curl -O https://norvig.com/ngrams/shakespeare_input.txt\n", - "! wc shakespeare_input.txt \n", - "# Print the number of lines, words, and characters" + "# Fetch the file if it does not already exist here\n", + "! [ -f shakespeare_input.txt ] || curl -O https://norvig.com/ngrams/shakespeare_input.txt " ] }, { @@ -65,6 +50,8 @@ "name": "stdout", "output_type": "stream", "text": [ + "4,573,338 characters and 832,301 words:\n", + "\n", "First Citizen:\n", "Before we proceed any further, hear me speak.\n", "\n", @@ -72,24 +59,29 @@ "Speak, speak.\n", "\n", "First Citizen:\n", - "You are all resolved rather to die than to famish?\n" + "You are all resolved rather to die than to famish?\n", + "\n", + "All:\n", + "Resolved. resolved.\n", + "\n", + "First Citizen:\n", + "First, you...\n" ] } ], "source": [ - "! head -8 shakespeare_input.txt \n", - "# First 8 lines" + "shakespeare: str = open(\"shakespeare_input.txt\").read()\n", + "\n", + "print(f'{len(shakespeare):,d} characters and {len(shakespeare.split()):,d} words:\\n\\n{shakespeare[:200]}...')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Python Code for n-Gram Model\n", + "## Python Code for *n*-Gram Language Model\n", "\n", - "I do some imports and then define two data types:\n", - "- A `Token` is an individual unit of text, a string of one or more characters.\n", - "- A `LanguageModel` is a subclass of `defaultdict` that maps a history of length *n* tokens to a `Counter` of the number of times each token appears immediately following the history in the training data." + "I'll start with some imports and simple definitions:" ] }, { @@ -100,25 +92,24 @@ "source": [ "import random\n", "from typing import *\n", - "from collections import defaultdict, Counter\n", + "from collections import defaultdict, Counter, deque\n", "\n", - "Token = str # Datatype to represent a token\n", + "type Token = str # Datatype to represent a token (a character or word)\n", "\n", - "class LanguageModel(defaultdict): \n", - " \"\"\"A mapping of {'history': Counter(next_token)}.\"\"\"\n", - " def __init__(self, n: int):\n", - " self.order = n\n", - " super().__init__(Counter)" + "cat = ''.join # Function to concatenate strings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "I define two main functions that do essentially all the work:\n", - "\n", - "- `train_LM` takes a sequence of tokens (the training data) and an integer *n*, and builds a language model of order *n*, formed by counting the times each token *t* occurs and storing that under the entry for the history *h* of *n* tokens that precede *t*. \n", - "- `generate_tokens` generates a random sequence of tokens, given an (optional) start sequence of tokens. At each step it looks at the history of previously generated tokens and chooses a new token at random from the language model's counter for that history." + "Now I define the class `LanguageModel`:\n", + "- A `LanguageModel` is a subclass of `defaultdict` that maps a history of length *order* tokens to a `Counter` of next tokens.\n", + " - The tokens in the history are concatenated together into one string to form the keys of the LanguageModel.\n", + "- The `__init__` method sets the order of the model and optionally accepts tokens of training data. \n", + "- The `train` method builds up the `{history: Counter(next_token)}` mapping from the training data.\n", + "- The `generate` method random samples `length` tokens from the mapping. \n", + "- The `gen` method is a convenience function to call `generate` and print the results." ] }, { @@ -127,32 +118,42 @@ "metadata": {}, "outputs": [], "source": [ - "def train_LM(tokens, order: int) -> LanguageModel:\n", - " \"\"\"Create and train a language model of given order on the given tokens.\"\"\"\n", - " LM = LanguageModel(order)\n", - " history = []\n", - " for token in tokens:\n", - " LM[cat(history)][token] += 1\n", - " history = (history + [token])[-order:] \n", - " return LM\n", + "class LanguageModel(defaultdict): \n", + " \"\"\"A mapping of {'history': Counter(next_token)}.\"\"\"\n", + " \n", + " def __init__(self, order: int, tokens=()):\n", + " \"\"\"Set the order of the model, and optionally initialize it with some tokens.\"\"\"\n", + " self.order = order\n", + " self.default_factory = Counter # Every history entry has a Counter of tokens\n", + " self.train(tokens)\n", "\n", - "def generate_tokens(LM: LanguageModel, length=1000, start=()) -> List[Token]:\n", - " \"\"\"Generate a random text of `length` tokens, with an optional start, from `LM`.\"\"\"\n", - " tokens = list(start)\n", - " while len(tokens) < length:\n", - " history = cat(tokens[-LM.order:])\n", - " tokens.append(random_sample(LM[history]))\n", - " return tokens" + " def train(self, tokens):\n", + " \"\"\"Go through the tokens, building the {'history': Counter(next_tokens)} mapping.\"\"\"\n", + " history = deque(maxlen=self.order) # History keeps at most `order` tokens\n", + " for token in tokens:\n", + " self[cat(history)][token] += 1\n", + " history.append(token)\n", + " return self\n", + "\n", + " def generate(self, length=1000, start=()) -> List[Token]:\n", + " \"\"\"Generate a random text of `length` tokens, from a sequence of `start` tokens.\n", + " At each step, consider the previous `self.order` tokens and randomly sample the next token.\"\"\"\n", + " tokens = list(start)\n", + " while len(tokens) < length:\n", + " history = cat(tokens[-self.order:])\n", + " tokens.append(random_token(self[history]))\n", + " return tokens\n", + "\n", + " def gen(self, length=1000, start=()) -> None:\n", + " \"\"\"Call generate and print the resulting tokens.\"\"\"\n", + " print(cat(self.generate(length, start)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Here are three auxiliary functions:\n", - "- `gen` is a convenience function to call `generate_tokens`, concatenate the resulting tokens, and print them.\n", - "- `random_sample` randomly chooses a single token from a Counter, with probability in proportion to its count.\n", - "- `cat` is a utility function to concatenate strings (tokens) into one big string." + "We'll need a function to randomly select a next token from one of the model's Counters:" ] }, { @@ -161,22 +162,16 @@ "metadata": {}, "outputs": [], "source": [ - "def gen(LM: LanguageModel, length=1000, start=()) -> None:\n", - " \"\"\"Call generate_tokens and print the resulting tokens.\"\"\"\n", - " print(cat(generate_tokens(LM, length, start)))\n", - " \n", - "def random_sample(counter: Counter) -> Token:\n", - " \"\"\"Randomly sample a token from the counter, proportional to each token's count.\"\"\"\n", - " return random.choices(list(counter), weights=list(counter.values()), k=1)[0]\n", - "\n", - "cat = ''.join # Function to join strings together" + "def random_token(counter: Counter) -> Token:\n", + " \"\"\"Randomly sample a token from a Counter, with probability proportional to each token's count.\"\"\"\n", + " return random.choices(list(counter), weights=list(counter.values()), k=1)[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Let's train a character-level language model of order 4 on the Shakespeare data. We'll call the model `LM4`. (Note that saying `tokens=data` means that the sequence of tokens is equal to the sequence of characters in `data`; in other words each character is a token.)" + "Let's train a character-level language model of order 4 on the Shakespeare data. We'll call the language model `LM`:" ] }, { @@ -185,16 +180,14 @@ "metadata": {}, "outputs": [], "source": [ - "data = open(\"shakespeare_input.txt\").read()\n", - "\n", - "LM = train_LM(tokens=data, order=4)" + "LM = LanguageModel(4, shakespeare)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Here are some examples of what's in the model, for various 4-character histories:" + "Here are some examples of what's in the model:" ] }, { @@ -217,15 +210,74 @@ "LM[\"chai\"]" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So `\"chai\"` is followed by either `'n'` or `'r'`. In contrast, almost any letter can follow `\"the \"`:" + ] + }, { "cell_type": "code", "execution_count": 8, - "metadata": {}, + "metadata": { + "collapsed": false, + "jupyter": { + "outputs_hidden": false + } + }, "outputs": [ { "data": { "text/plain": [ - "'n'" + "Counter({'s': 2058,\n", + " 'w': 1759,\n", + " 'c': 1561,\n", + " 'm': 1392,\n", + " 'p': 1360,\n", + " 'f': 1258,\n", + " 'b': 1217,\n", + " 'd': 1170,\n", + " 't': 1109,\n", + " 'g': 1037,\n", + " 'h': 1029,\n", + " 'l': 1006,\n", + " 'r': 804,\n", + " 'k': 713,\n", + " 'e': 704,\n", + " 'n': 616,\n", + " 'a': 554,\n", + " 'o': 530,\n", + " 'v': 388,\n", + " 'i': 298,\n", + " 'q': 171,\n", + " 'D': 146,\n", + " 'y': 120,\n", + " 'u': 105,\n", + " 'L': 105,\n", + " 'F': 103,\n", + " 'T': 102,\n", + " 'j': 99,\n", + " 'C': 81,\n", + " 'E': 77,\n", + " 'G': 75,\n", + " 'M': 54,\n", + " 'P': 54,\n", + " 'R': 45,\n", + " 'S': 40,\n", + " 'B': 31,\n", + " 'J': 30,\n", + " 'A': 29,\n", + " 'K': 22,\n", + " 'H': 20,\n", + " 'V': 18,\n", + " 'N': 18,\n", + " 'I': 14,\n", + " 'W': 14,\n", + " \"'\": 10,\n", + " 'Q': 7,\n", + " 'z': 6,\n", + " 'O': 3})" ] }, "execution_count": 8, @@ -234,7 +286,16 @@ } ], "source": [ - "random_sample(LM[\"chai\"])" + "LM[\"the \"]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generating Shakespeare\n", + "\n", + "We cann generate a random text from the order 4 model:" ] }, { @@ -248,81 +309,53 @@ }, "outputs": [ { - "data": { - "text/plain": [ - "Counter({'p': 1360,\n", - " 's': 2058,\n", - " 'l': 1006,\n", - " 'o': 530,\n", - " 'g': 1037,\n", - " 'c': 1561,\n", - " 'a': 554,\n", - " 'C': 81,\n", - " 'r': 804,\n", - " 'h': 1029,\n", - " 'R': 45,\n", - " 'd': 1170,\n", - " 'w': 1759,\n", - " 'b': 1217,\n", - " 'm': 1392,\n", - " 'v': 388,\n", - " 't': 1109,\n", - " 'f': 1258,\n", - " 'i': 298,\n", - " 'n': 616,\n", - " 'V': 18,\n", - " 'e': 704,\n", - " 'u': 105,\n", - " 'L': 105,\n", - " 'y': 120,\n", - " 'A': 29,\n", - " 'H': 20,\n", - " 'k': 713,\n", - " 'M': 54,\n", - " 'T': 102,\n", - " 'j': 99,\n", - " 'q': 171,\n", - " 'K': 22,\n", - " 'D': 146,\n", - " 'P': 54,\n", - " 'S': 40,\n", - " 'G': 75,\n", - " 'I': 14,\n", - " 'B': 31,\n", - " 'W': 14,\n", - " 'E': 77,\n", - " 'F': 103,\n", - " 'O': 3,\n", - " \"'\": 10,\n", - " 'z': 6,\n", - " 'J': 30,\n", - " 'N': 18,\n", - " 'Q': 7})" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" + "name": "stdout", + "output_type": "stream", + "text": [ + "First Fishes come, trouble annot have the here reign's every madness, it is repart on hath than of that were is time little be faint of that came of a monstands his on and the wonderstandiscoverfluous nest ask again! thou should writ than we, I'll his good Mercules; your\n", + "sonneur, my good is no make me, yet were here is very us;\n", + "And, nobler the more at me not his preport,\n", + "Such moved on:\n", + "But not by my duke\n", + "To business: pleasure no moral bed.\n", + "Harry noble an end\n", + "Do more that were I have do behind,\n", + "I go to judgment, and he as he,'that I have come.\n", + "Julius, and Penthorough at lame you helps as this for, 'tis not\n", + "right i' the earth Boling it fing, sir.\n", + "\n", + "HAMLET:\n", + "All:\n", + "Their me,\n", + "And yet you speak; I strong;\n", + "But were subject his his pride up a throw\n", + "One way;\n", + "Still ye well me an enemy,\n", + "I will sick, cause,\n", + "And cut of they saithful necess, if God!\n", + "For than you how approves compound in Gloucester tribute\n", + "A grave?\n", + "\n", + "BURGUNDY:\n", + "Thy good with lessenger with done is\n", + "own deep; ha!\n", + "Will meat great lady troubl\n" + ] } ], "source": [ - "LM[\"the \"]" + "LM.gen()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "So `\"chai\"` is followed by either `'n'` or `'r'`, and almost any letter can follow `\"the \"`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Generating Shakespeare\n", + "Order 4 captures the structure of plays, mentions some characters, and generates mostly English words. But the words don't always go together to form grammatical sentences, and there is certainly no coherence or plot. \n", "\n", - "We cann generate a random text from the order 4 character model:" + "## Generating Order 7 Shakespeare\n", + "\n", + "What if we increase the model to order 7? Or 10? The output gets a bit better, roughly as good as the RNN models that Karpathy shows, and all from a much simpler *n*-gram model." ] }, { @@ -339,67 +372,67 @@ "name": "stdout", "output_type": "stream", "text": [ - "First, hypocrity.\n", + "First Citizen:\n", + "Neighbours shall we in our enemies.\n", "\n", - "Messenge a bear she is, malice for the people's lion!\n", + "First Lord:\n", + "Behind this crown'd, Laertes,\n", + "Will your sake, speak of African;\n", + "Where is three.\n", "\n", - "TALBOT:\n", - "Thou dwell and,\n", - "And liest felt malice, by Cleon, then, sir?\n", + "BIRON:\n", + "Why, I prithee, take me mad:\n", + "Hark, how our company.\n", "\n", - "QUEEN:\n", - "Yes, we press might\n", - "It may purch:\n", - "A coward and Valenting kiss,\n", - "And perils house,\n", - "Till deeds as dost ther;\n", - "And I, to keep hour could nevership not yet, I this cause.\n", + "MISTRESS FORD:\n", "\n", - "Solicy.\n", + "All:\n", + "Our dukedoms.\n", + "The break with\n", + "you.\n", "\n", - "NYM:\n", - "To his not my lord\n", - "'her beggars never hear us\n", - "to do\n", - "Not sell with the now I furthen here that thou all.\n", + "LEONTES:\n", + "Say you this true at first inducement.\n", "\n", - "KING RICHARD II:\n", - "Go, but boy lords this kins\n", - "Answer unkindness.'\n", - "Plant reason\n", - "Are mayst pleasure\n", - "What cousiness come inst the devils, and I say, his spurn by serves;\n", - "For and upon't her than lord.\n", + "IAGO:\n", + "There is set for't! And now my deeds be grieves her question.\n", "\n", - "QUEEN:\n", - "Then, nothing of he will leaving.\n", + "LEONATO:\n", + "Brother with thy state, and here together with usurping her still\n", + "And looks, bid her contemn'd revolt: this night;\n", + "For, in his tears not found the players cannot, take this.\n", "\n", - "SIR TOBY BELCH:\n", - "Come of an even to portculling end in are a dog; and thither with grief, heart is much vicion! By their sick with\n", - "corrupt,\n", - "But my lords: but home\n", - "Are than and shot as the once inter way I\n", - "shall,\n", - "This fruit.\n", + "CADE:\n", "\n", - "LONGAVILLES:\n", - "'Tis yet at hatility the eachelor our plead me, the cram with mattended grace and my yet \n" + "DICK:\n", + "My heart that in this, till thou thus, sir. Fare you were so;\n", + "To disprove to hear from the man.\n", + "\n", + "ARIEL:\n", + "I pray you, the gates;\n", + "And makes a sun and disjoin'd penitent head of thine;\n", + "With fear; my master, sir, no; the presupposed\n", + "Upon the fiend's wrong\n", + "And fertile land of the quern\n", + "And ruminate tender'd herself: he shall stay at home;\n", + "And chides wrong.\n", + "I will we bite our castle:\n", + "She died,\n", + "That were out by the painful, and \n", + "CPU times: user 2.48 s, sys: 46.8 ms, total: 2.53 s\n", + "Wall time: 2.53 s\n" ] } ], "source": [ - "gen(LM)" + "%time LanguageModel(7, shakespeare).gen()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Order 4 captures the structure of plays, mentions some characters, and generates mostly English words. But the words don't always go together to form grammatical sentences, and there is certainly no coherence or plot. \n", - "\n", - "## Generating Order 7 Shakespeare\n", - "\n", - "What if we increase the model to order 7? Or 10? The output gets a bit better, roughly as good as the RNN models that Karpathy shows, and all from a much simpler *n*-gram model." + "## Generating Order 10 Shakespeare" ] }, { @@ -416,127 +449,94 @@ "name": "stdout", "output_type": "stream", "text": [ - "First Clown:\n", - "What are so envenom'd: therefore, good Ursula\n", - "Walk in thy opinion and hit\n", - "The cheater: but lechery eats in my person's sacred king.\n", - "Ay me! sad hours must fight no more.\n", + "First Citizen:\n", + "Woe to the hearts of men\n", + "The thing I am forbid;\n", + "Or study where I had and have it; and much more ease; for so I have.\n", "\n", - "OLIVIA:\n", - "Are you think you; you well:\n", - "A gallant ship, sir; you have not denies\n", - "The discourse our supposed\n", - "He that makes King Edward from our counsel in my close they will this young daughter is my birth;\n", - "But no man have I here resignation of them: while apart.\n", - "Stand in this I challenge much as to mince nought;\n", - "Watch'd you,\n", - "Your displeasures, and all the heart bleed at Plashy too;\n", - "My operant power unto Octavius' tent\n", - "How the hand\n", - "Of him that I were you venture of muttons, be your leave,\n", - "For which the anvil of him; you shall acquitted\n", - "Without book are adventurously bound together.\n", + "Second Lord:\n", + "He had no other death concludes but what thou wilt.\n", + "How now, Simple! where have you taste of thy abhorr'd ingredients of our loves again,\n", + "Alike betwitched by the Frenchman his companion of the gesture\n", + "One might interpreter, you might pardon him, sweet father, do you here? things that I bought mine own.\n", "\n", - "FORD:\n", - "Marry, we have a parish-top. What is the matters. This curse thought of revenge,\n", - "For governed from the proclaim it civil dissemblies to trust not colour will.\n", + "KING RICHARD II:\n", + "A lunatic lean-witted fools\n", + "The way twice o'er, I'll weep. O fool, I shall be publish'd, and\n", + "Her coronation-day,\n", + "When Bolingbroke ascends my throne of France upon his sudden seem,\n", + "I would be the first house, our story\n", + "What we have o'erheard\n", + "Your royal grace!\n", "\n", - "TITANIA:\n", - "My good king, from the very know, sir king of her lips,\n", - "\n" + "DUKE VINCENTIO:\n", + "Have after. To what end he gave me fresh garments must not then respective lenity,\n", + "And all-to topple: pure surprised:\n", + "Guard her till this osier cage of ours\n", + "Were nice and continents! what mutiny!\n", + "What raging of their emperor\n", + "And to conclude to hate me.\n", + "\n", + "KI\n" ] } ], "source": [ - "gen(train_LM(data, order=7))" + "LanguageModel(10, shakespeare).gen()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Generating Order 10 Shakespeare" + "## Probabilities and Smoothing\n", + "\n", + "Sometimes we'd rather see probabilities, not raw counts. Given a language model `LM`, the probability *P*(*token* | *history*) could be computed as follows:" ] }, { "cell_type": "code", "execution_count": 12, - "metadata": { - "collapsed": false, - "jupyter": { - "outputs_hidden": false - } - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "First Citizen:\n", - "We are blest in peace and honour than\n", - "Your gates against an alien\n", - "That by direct or by collateral hand\n", - "They find us touch'd, or carved to thee.\n", - "\n", - "CARDINAL WOLSEY:\n", - "Your grace hath blessed and engaged to many Greeks,\n", - "Even in these cases, where in gore he lay insteep'd,\n", - "And take it.\n", - "\n", - "PROTEUS:\n", - "My gracious king:\n", - "And I do wish\n", - "That your pains the hire;\n", - "If you do wrong you? alas, our places.\n", - "\n", - "SATURNINUS:\n", - "Why, worthy Margaret, that is the mad mothers than snow,\n", - "And all the posterns\n", - "Clear them out.\n", - "\n", - "All:\n", - "A heavy reckoning to make\n", - "Mine eyes too, examined my parts with death, goodman Dull.\n", - "\n", - "DULL:\n", - "Which is not confess she does: there's another place\n", - "And find me well deliver:\n", - "Mark Antony, she pursed up\n", - "his heart think it cites us, brother, I go; I'll win them, fear it not;\n", - "Let not thy nature; let not every man's Hero:\n", - "\n", - "CLAUDIO:\n", - "I know him; 'tis a mere scutcheon: and so\n", - "ends my catechism.\n", - "\n", - "EARL OF DOUGLAS:\n", - "'Faith, that hath gull'd thee there.\n", - "But, room, fairy! here comes your father's \n" - ] - } - ], + "metadata": {}, + "outputs": [], "source": [ - "gen(train_LM(data, order=10))" + "def P(token, history, LM: LanguageModel): \n", + " \"The probability that token follows history.\"\"\"\n", + " return LM[history][token] / sum(LM[history].values())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Aside: Probabilities and Smoothing\n", - "\n", - "Sometimes we'd rather see probabilities, not raw counts. Given a language model `LM`, the probability *P*(*t* | *h*) can be computed as follows:" + "What's the probaility that the letter \"n\" follows the four letters \"chai\"?" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "0.6902654867256637" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "def P(t, h, LM: LanguageModel): \n", - " \"The probability that token t follows history h.\"\"\"\n", - " return LM[h][t] / sum(LM[h].values())" + "P('n', 'chai', LM)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "What about the letter \"s\"?" ] }, { @@ -547,7 +547,7 @@ { "data": { "text/plain": [ - "0.09286165508528112" + "0.0" ] }, "execution_count": 14, @@ -556,27 +556,33 @@ } ], "source": [ - "P('s', 'the ', LM)" + "P('s', 'chai', LM)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Shakespeare never wrote about \"chaise longues,\" or \"chai tea\" so the probability of an \"s\" or space following \"chai\" is zero, according to our language model. But do we really want to say it is absolutely impossible for the sequence of letters \"chais\" or \"chai \" to appear anywhere in a text, just because we didn't happen to see it in our training data? More sophisticated language models use [**smoothing**](https://en.wikipedia.org/wiki/Kneser%E2%80%93Ney_smoothing) to assign non-zero (but small) probabilities to previously-unseen sequences. A simple type of smoothing is \"add-one smoothing\"; it assumes that if we have counted *N* tokens following a given history, then the probability of an unseen token is 1 / (*N* + 1), and the probabilities for the previously-seen tokens are reduced accordingly (dividing by *N* + 1 instead of *N*):" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "0.6902654867256637" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ - "P('n', 'chai', LM)" + "def P(t, h, LM: LanguageModel): \n", + " \"The probability that token t follows history h, using add-one smoothing.\"\"\"\n", + " N = sum(LM[h].values())\n", + " return max(1, LM[h][t]) / (N + 1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "That gives us:" ] }, { @@ -587,7 +593,7 @@ { "data": { "text/plain": [ - "0.30973451327433627" + "0.008771929824561403" ] }, "execution_count": 16, @@ -595,71 +601,44 @@ "output_type": "execute_result" } ], - "source": [ - "P('r', 'chai', LM)" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "0.0" - ] - }, - "execution_count": 17, - "metadata": {}, - "output_type": "execute_result" - } - ], "source": [ "P('s', 'chai', LM)" ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "0.0" + "0.6842105263157895" ] }, - "execution_count": 18, + "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "P(' ', 'chai', LM)" + "P('n', 'chai', LM)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Shakespeare never wrote about \"chaise longues,\" or \"chai tea\" so the probability of an `'s'` or `' '` following `'chai'` is zero, according to our language model. But do we really want to say it is absolutely impossible for the sequence of letters `'chais'` or `'chai '` to appear in a generated text, just because we didn't happen to see it in our training data? More sophisticated language models use [**smoothing**](https://en.wikipedia.org/wiki/Kneser%E2%80%93Ney_smoothing) to assign non-zero (but small) probabilities to previously-unseen sequences. In this notebook we stick to the basic unsmoothed model." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Aside: Starting Text\n", + "## Starting Text\n", "\n", - "One thing you may have noticed: all the generated passages start the same. Why is that? Because the training data happens to start with the line \"First Citizen:\", and so when we call `generate_tokens`, we start with an empty history, and the only thing that follows the empty history in the training data is the letter \"F\", the only thing that follows \"F\" is \"i\", and so on, until we get to a point where there are multiple choices. We could get more variety in the start of the generated text by breaking the training text up into multiple sections, so that each section would contribute a different possible starting point. But that would require some knowledge of the structure of the training text; right now the only assumption is that it is a sequence of tokens/characters.\n", + "One thing you may have noticed: all the generated passages start the word \"First\". Why is that? Because the training data happens to start with the line \"First Citizen:\", and so when we call `generate_tokens`, we start with an empty history, and the only thing that follows the empty history in the training data is the letter \"F\", the only thing that follows \"F\" is \"i\", and so on, until we get to a point where there are multiple choices. We could get more variety in the start of the generated text by breaking the training text up into multiple sections, so that each section would contribute a different possible starting point. But that would require some knowledge of the structure of the training text; right now the only assumption is that it is a sequence of tokens/characters.\n", "\n", "We can give a starting text to `generate_tokens` and it will continue from there. But since the models only look at a few characters of history (just 4 for `LM`), this won't make much difference. For example, the following won't make the model generate a story about Romeo:" ] }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 18, "metadata": {}, "outputs": [ { @@ -667,52 +646,31 @@ "output_type": "stream", "text": [ "ROMEO:\n", - "What swifter of so much a doubled,\n", - "And I may, the dare, by the lives done own rich, that you naughter-out appearancess of your turn us,\n", - "Whose heave born blow be prology.\n", - "Keeper: proud.\n", + "Is the rash they news, and\n", + "the night too her never, tie here queen,\n", + "And, in lady store known\n", + "in practises;\n", + "Whose thy master thought;\n", + "Then for my rose, for nothese him that honour body and I now nights have to make with noted.\n", "\n", - "BALTHAZAR:\n", - "Let so shall a picture we mercy,\n", - "And the king this bondage an e'er that I storator:\n", - "\n", - "FERDINAL CAMPEIUS:\n", - "First Servant:\n", - "I this on your end\n", - "Louder with methink I do bid it: if heavens:\n", - "shall this vest; and run, my troops then, good and so forty wife.\n", - "Invited.\n", - "\n", - "MACBETH:\n", - "Avaunt,\n", - "And Here is days.\n", - "\n", - "ORLEANS:\n", - "And friend;\n", - "Dismissing, she detain the time thou scourselves itself:\n", - "but with the gent the half abused!\n", - "\n", - "QUEEN:\n", - "Their daughteous most in my lost fight\n", - "The prey to say you give himselves Lord's the comes bold your look the lady, look you, Fabian:\n", - "A pass!\n", - "\n", - "EMILIA:\n", - "Is heater is they my him; and chard for my lord, ripe to Pompey, I and let us have your woo?\n", - "\n", - "Think upon: indeed? Do noison.\n", - "'Tis most in a suddenly out, sir; you are\n", - "A cist\n", - "For noble Antent our\n", - "daughter disclose him.\n", - "\n", - "CELIA:\n", - "Dou\n" + "ANTIPHOLUS OF EPHEN SCROOP:\n", + "Gloucested, and dote: marry, were cours, I deputes our this, and hurt was. Yet, could probations, I hear me, Rosal to chard,\n", + "Which thy error offer'd you parce Will bears,\n", + "The nature and how prever my mont: after come to bear the that worthly to Cyprus.--Help, ho! how\n", + "did:\n", + "If guard this daughter,\n", + "Were up ruled for sister, be hour, than with mouth, my patient you my hence Helent denuncle, la, had for thank the farth\n", + "The gate:\n", + "My prither business. He behold, in would not ruins. What, assage of fles, that were this,\n", + "How it is not thence; for the matter. Thaisanio;\n", + "Why, where Christial of a bay,\n", + "But of steeds the not me save you owest be dog-day,\n", + "His slain If, a black, you die against the compey, of thine own the riots do lustry, may\n" ] } ], "source": [ - "gen(LM, start='ROMEO')" + "LM.gen(start='ROMEO')" ] }, { @@ -728,7 +686,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 19, "metadata": { "collapsed": false, "jupyter": { @@ -751,7 +709,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 20, "metadata": { "collapsed": false, "jupyter": { @@ -772,6 +730,173 @@ "We'll start with an order-10 character model, and compare that to an order-20 model. We'll generate a longer text, because sometimes 1000 characters ends up being just one long comment." ] }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "collapsed": true, + "jupyter": { + "outputs_hidden": true + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/*\n", + " * linux/kernel.h>\n", + "#include \n", + "#include \n", + "\n", + "struct snapshot_device_available space if\n", + " * such as list traversal.\n", + " */\n", + "struct sigpending *list, siginfo_t info;\n", + "\tif (!ret)\n", + "\t\trb->aux_priv = event->pmu = pmu;\n", + "\n", + "\treturn BUF_PAGE_SIZE(slots)\t\t\t\\\n", + "\t(offsetof(struct printk_log) + msg->text_len;\n", + "\tif (path)\n", + "\t\taudit_log_n_untrustedstring(ab, \"remove_rule\");\n", + "\t\t\tlist_add(&nt->entry, list);\n", + "\tcpu_buffer->irq_work.work, rb_wake_up_waiters - wake up the first child if it\n", + "\t\t * is tracked on a waitqueue_head_t *bit_waitqueue_head_init(&rt_rq->rt_nr_boosted = 0;\n", + "\tret = test_jprobe();\n", + "\tif (err)\n", + "\t\t\tgoto process\n", + " * @len: max length to calculate_period(event);\n", + "\tprint_ip_sym(s, *p, flags);\n", + "\tvoid (*write_delay,\n", + "\t.writeunlock\t= torture_random(trsp) % (cxt.nrealwriters_stress = cxt.nrealwriters_stress >= 0)\n", + "\t\t\tmark_reg_unknown_value(regs);\n", + "\n", + "\tcurrent->lockdep_depth = curr->lockdep_depth; i++) {\n", + "\t\tif (KDB_FLAG(CMD_INTERRUPT)) {\n", + "\t\t/* We need to keep it from\n", + "\t\t * the current thread will park without risk of ENOMEM.\n", + " */\n", + "struct cgroup *cgrp;\n", + "\n", + "\tif (kprobe_unused(&op->kp));\n", + "\t\tlist_del(&use->target->name);\n", + "\t\tif (ret) {\n", + "\t\tpr_info(\"%sconsole [%s%d] disabled\\n\",\n", + "\t\t\t\t\t\tt_sector(ent),\n", + "\t\t\t MAJOR(n->rdev));\n", + "\tif (!new_base->cpu_base->clock_base =\n", + "\t{\n", + "\t\t{\n", + "\t\t\t.index = HRTIMER_NORESTART;\n", + "}\n", + "\n", + "void task_numa_env *env)\n", + "{\n", + "\tif (autogroup_kref_put(prev);\n", + "}\n", + "\n", + "static void ptrace_notify(int signr)\n", + "{\n", + "\tstruct module *mod = list_entries); /* Out-of-bounds values?\n", + "\t */\n", + "\tfor (i = 0; i < __nenv; i++) {\n", + "\t\t\t\tif (s2->pid == pid)\n", + "\t\treturn;\n", + "\t}\n", + "\n", + "\tif (attr == &dev_attr_current_device *dev, void *regs)\n", + "{\n", + "\tint oldstate;\n", + "\tint\t\t\tindex;\n", + "};\n", + "\n", + "/**\n", + " * platform_mode)\n", + "{\n", + "\tif (p->policy == SETPARAM_POLICY)\n", + "\t\tpolicy = policy;\n", + "\n", + "\tif (dl_policy(int policy)\n", + "{\n", + "\treturn -ENODEV;\n", + "\n", + "\treturn 1;\n", + "\t\t}\n", + "\t}\n", + "\n", + "\ti = (unsigned long pid;\n", + "\t\tif (get_user(parser, ubuf, cnt)) {\n", + "\t\tINIT_LIST_HEAD(&event->list, list) {\n", + "\t\tif (context->proctitle.len;\n", + "out:\n", + "\tmutex_unlock(&autosleep_init(void)\n", + "{\n", + "\trcu_lockdep_assert_held(&env->src_rq, as\n", + " * part of the buffer pages */\n", + "\tINIT_LIST_HEAD(&image->dest_pages, lru) {\n", + "\t\taddr = last_addr;\n", + "\t\tradix = last_radix = radix;\n", + "\n", + "\tif (bytes == 0)\n", + "\t\treturn 0;\n", + "\n", + "\ts = pending->signal.sig;\n", + "\tm = mask->sig;\n", + "\n", + "\t/*\n", + "\t * It is possible to record\n", + " * @iter: The iterator in this case we don't even try to\n", + "\t * accumalator (@total) and @count\n", + " */\n", + "\n", + "static const struct filter_pred_fn_t) (struct timespec __user *rmtp = restart->nanosleep.expires);\n", + "\t\t\tbreak;\n", + "\t\t\tprintk_deferred(\" PADATA_CPU_PARALLEL;\n", + "\tret = 0;\n", + "\t\t\tif (count >= PAGE_SIZE);\n", + "\n", + "#define TRACE_FTRACE_MAX);\n", + "\tif (!pd->squeue, cpu);\n", + "\t}\n", + "}\n", + "\n", + "static int cgroup_wq_init(void)\n", + "{\n", + "#if defined(CONFIG_RCU_FANOUT_3\n", + "# define NUM_RCU_LVL_3 + NUM_RCU_LVL_4\t 0\n", + "#endif\n", + "\n", + "/* Location of the percpu refcnts are configured by writing them to write to.\n", + " */\n", + "\n", + "static int min_offline = delta;\n", + "\t\t}\n", + "\t} while (sg != env->sd->nr_balance_failed)\n", + "\t\tforce_sig_info(sig, info, p, 1, 0);\n", + "\t\t\tgoto err;\n", + "#endif\n", + "\n", + "\traw_spin_lock_init(void)\n", + "{\n", + "\ttrace_latency_header(s);\n", + "}\n", + "#else\n", + "static int __init setup_relax_\n" + ] + } + ], + "source": [ + "LanguageModel(10, linux).gen(length=3000)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Order 20 C++" + ] + }, { "cell_type": "code", "execution_count": 22, @@ -787,303 +912,130 @@ "output_type": "stream", "text": [ "/*\n", - " * linux/kernel.h>\n", - "#include \n", - "#include irq_unmask\t= noop,\n", - "\t.irq_enable)\n", - "\t\tclear_frozen = false;\n", - "\n", - "\tif (time_before(end_time, timer_t, timer_id, &flag);\n", - "\tif (err)\n", - "\t\treturn;\n", - "\n", - "\ttracing_resize_ring_buffer_record_disabled)\n", - "\t\treturn;\n", - "\tif (desc) {\n", - "\t\traw_spin_lock_irqsave(&tracepoint_str + (*pos - 1);\n", - "\ti++;\n", - "\n", - "find_first_elem;\n", - "\t}\n", - "\n", - "\t/* sort them */\n", - "\tarray_desc\n", - "#define CAP_PI\t\t(void *)rec->ip);\n", - "\t\t\t/* Ftrace is shutting down system\");\n", - "\tshutdown_task != NULL)\n", - "\t\tevent->rcu_pending()) {\n", - "\t\terr = -EINVAL;\n", - "}\n", - "\n", - "/* Special cases that /proc allows\n", - "\t * for built-in exception is the concept of\n", - "a \"sequence count */\n", - "\t\titer->trace is a kernel\n", - " * and userspace address.\n", - "\t\t * Either we have now waited for the first call.\n", - " * Return: %false if it was a timeout or signal will be freed in case of UMH_NO_WAIT)\t/* task has the given prefix.\n", - " *\t0 if no string for all tasks in the system configuration.\n", + " * Copyright (C) 2009 Red Hat, Inc., Ingo Molnar \n", " *\n", - " * Use this only\n", - " * one call to synchronized by task_work_add()\n", + " * Contributors at various stages not listed above:\n", + " * Jason Wessel ( jason.wessel@windriver.com>\n", " *\n", - " * Flush any pending */\n", - "\tdiv = div_s64(offset64 << NTP_SCALE_SHIFT. */\n", - "extern unsigned long msleep_interruptible(\n", - "\t\t\trnp->nocb_gp_tail;\n", - "\tbool nocb_leader = rdp_spawn)\n", - "\t\t\t\trdp_last = rdp;\n", - "\t\t\t\trdp = rdp->nocb_follower_head;\n", - "}\n", - "\n", - "/*\n", - " * RCU global state. */\n", - "\n", - "/*\n", - " * Priority Inheritance state:\n", + " * Copyright (C) 1992, 1998-2006 Linus Torvalds, Ingo Molnar\n", + " *\n", + " * This file contains macros used solely by rtmutex.c. Debug version.\n", " */\n", - "struct task_struct *p, int user_namespace *user_ns, struct task_struct() in task_numa_env env = {\n", - "\t\t.p = p,\n", "\n", - "\t\t.src_cpu = busiest;\n", + "extern void\n", + "rt_mutex_deadlock_account_lock(lock, proxy_owner);\n", + "\trt_mutex_set_owner(lock);\n", + "\t\tmutex_acquire(&lock->dep_map, subclass, 0, nest_lock, ip);\n", "\n", - "out_balanced:\n", - "\t\t/*\n", - "\t\t * If the scheduled. */\n", - "\tMULTI_STOP_PREPARE, cpu);\n", - "\tBUG_ON(event->parent)\n", - "\t\tgoto out;\n", + "\tif (mutex_optimistic_spin(struct mutex *lock, struct ww_acquire_ctx *ctx)\n", + "{\n", + "\tstruct mutex_waiter *waiter);\n", + "extern void debug_rt_mutex_init_waiter(&waiter);\n", + "\tRB_CLEAR_NODE(&rt_waiter.tree_entry);\n", + "\n", + "\traw_spin_lock(&this_rq->lock);\n", + "\n", + "\tupdate_blocked_averages(cpu);\n", + "\n", + "\trcu_read_lock();\n", + "\tp = find_task_by_vpid(pid);\n", + "\t\tif (p)\n", + "\t\t\terr = posix_cpu_clock_get,\n", + "\t.timer_create\t= pc_timer_create,\n", + "\t.timer_set\t= pc_timer_settime,\n", + "\t.timer_del\t= pc_timer_delete,\n", + "\t.timer_get\t= pc_timer_gettime,\n", + "};\n", + "#ifndef _CONSOLE_CMDLINE_H\n", + "#define _CONSOLE_CMDLINE_H\n", + "\n", + "struct console_cmdline\n", + "{\n", + "\tchar\tname[16];\t\t\t/* Name of the driver\t */\n", + "\tint\tindex;\t\t\t\t/* Minor dev. to use\t */\n", + "\tchar\t*options;\t\t\t/* Options for braille driver */\n", + "#endif\n", + "};\n", + "\n", + "#endif\n", + "/* audit_watch.c -- watching inodes\n", + " *\n", + " * Copyright 2003-2007 Red Hat Inc., Durham, North Carolina.\n", + " * Copyright 2005 Hewlett-Packard Development Company, L.P.\n", + " *\n", + " * Authors: Waiman Long \n", + " */\n", + "#include \n", + "\n", + "/*\n", + " * Define shape of hierarchy based on NR_CPUS, CONFIG_RCU_FANOUT, and\n", + " * CONFIG_RCU_FANOUT;\n", + "\t} else {\n", + "\t\tint ccur;\n", + "\t\tint cprv;\n", + "\n", + "\t\tcprv = nr_cpu_ids;\n", + "\t\tfor (i = rcu_num_lvls - 1; i >= 0; i--) {\n", + "\t\tif (pipesummary[i] != 0)\n", + "\t\t\tbreak;\n", "\t}\n", - "\ttrace_access_lock, cpu);\n", - "\tatomic_inc(&buffer->mutex);\n", - "\tINIT_LIST_HEAD(&cset->mg_tasks);\n", - "\t\tfakewriters, 4, \"Number of page frames total] + PAGES_FOR_IO)) / 2\n", - "\t\t\t- 2 * DIV_ROUND_UP(NR_CPUS, RCU_FANOUT;\n", + "\n", + "\tpr_alert(\"%s%s \", torture_type, TORTURE_FLAG);\n", + "\tpr_cont(\"Free-Block Circulation: \");\n", + "\tfor (i = 0; i < TVN_SIZE; i++) {\n", + "\t\tmigrate_timer_list(new_base, old_base->tv1.vec + i);\n", + "\tfor (i = 0; i < length; i++) {\n", + "\t\t\tunsigned long rlim_rtprio =\n", + "\t\t\t\t\ttask_rlimit(p, RLIMIT_RTPRIO);\n", + "\n", + "\t\t\t/* can't set/change the rt policy */\n", + "\t\t\tif (policy != p->policy && !rlim_rtprio)\n", + "\t\t\t\treturn -EPERM;\n", + "\t\t}\n", + "\t}\n", + "\t/* nothing invalid, do the changes */\n", + "\tfor (i = 0; i < csn; i++) {\n", + "\t\tstruct cpuset *a = csa[i];\n", + "\t\tint apn = a->pn;\n", + "\n", + "\t\tfor (j = 0; j < chain->depth - 1; j++, i++) {\n", + "\t\t\tint lock_id = curr->held_locks[i].class_idx - 1;\n", + "\t\t\tchain_hlocks[chain->base + i];\n", + "}\n", + "\n", + "/*\n", + " * Look up a dependency chain. If the key is not present yet then\n", + " * add it and return 1 - in this case the new dependency does not connect a hardirq-safe\n", + "\t * lock with a hardirq-unsafe lock (in the full forwards-subgraph starting at :\n", + "\t */\n", + "\tif (!check_usage(curr, prev, next, bit,\n", + "\t\t\t exclusive_bit(bit), state_name(bit)))\n", + "\t\treturn 0;\n", + "\n", + "\tret = eligible_child(wo, p);\n", + "\tif (!ret)\n", + "\t\treturn 0;\n", "\n", "\t/*\n", - "\t * The taskstats *mk_reply(struct sched_rt_entity *rt_se)\n", - "{\n", - "\tstruct fd output;\n", - "\t\t\tret = blk_trace_getrq, NULL);\n", - "\tcd.wrap_kt, HRTIMER_MODE_ABS_PINNED, 0);\n", - "}\n", - "\n", - "/**\n", - " * ptrace_trap_notify(t);\n", - "\t\t\t}\n", - "\t\t\tif (later_mask = mask;\n", - "\tcurrent->curr_ret_stack, cpu);\n", - "\t}\n", - "\tpreempt_disable_cmds(void)\n", - "{\n", - "\treturn wl;\n", - "}\n", - "#else\n", - "static inline void module_unload_free(struct module *mod;\n", - "\n", - "\tmod = __module_get(dev->owner))\n", + "\t * Clockevents returns -ETIME, when the event was scheduled out.\n", + "\t */\n", + "\tif (ctx->task && cpuctx->task_ctx != ctx)\n", "\t\treturn -EINVAL;\n", "\n", - "\traw_spin_unlock(&cfs_b->lock\n", - " * must be able to reap\n", - "\t * it yet.\n", - "\t */\n", - "\tftrace_hash(new_hash);\n", - "\t\t}\n", - "\t\tmemory_bm_clear_bit(KTHREAD_IS_PARKED, &self->flags)) {\n", - "\t\tif (desc->wake_depth++ == 0) {\n", - "\t\t/* We should not be called unless tick_nohz_full_enabled() || rt_b->rt_runtime == period, ie unlimited)\\n\"\n", - "#endif\n", - "\t\"\\n trace_marker\\t\\t- Write 0/1 to enable function */\n", - "\tcur_stack >= env->prog->aux->ops->\n", - "\t\t\tconvert_ctx_access(int off, int size,\n", - "\t\t\t\tenv->flags &= ~mask;\n", - "}\n", + "\t/* same value, noting to do */\n", + "\tif (timer == pmu->hrtimer_interval_ms)\n", + "\t\treturn count;\n", "\n", - "static const struct cpupri *cp,\n", - "\t\t struct held_lock *hlock)\n", - "{\n", - "\treturn __wakeup_reset(tr);\n", + "\tpmu->hrtimer_interval_ms = timer;\n", "\n", - "\tset_tracer_flags->val;\n", - "\ttrace_on = tracing_open,\n", - "\t.read\t\t= seq_read,\n", - "\t.write\t\t= kgdb_console_write,\n", - "\t.seq_show = cpuset_read_u64,\n", - "\t\t.write_u64 = cpuset_write_res\n" + "\n" ] } ], "source": [ - "gen(train_LM(linux, order=10), length=3000)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Order 20 C++" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "collapsed": false, - "jupyter": { - "outputs_hidden": false - } - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/*\n", - " * linux/kernel/irq/proc.c\n", - " *\n", - " * Copyright (C) 2009 Jason Baron \n", - " * Copyright (C) 2009 Jason Baron \n", - " * Copyright (C) 2006 Rafael J. Wysocki \n", - " *\n", - " * This file is released under the GPLv2.\n", - " */\n", - "\n", - "#include \n", - "#include \n", - "#include \n", - "\n", - "#include \n", - "#include \n", - "\n", - "#include \"trace_probe.h\"\n", - "\n", - "#define KPROBE_EVENT_SYSTEM);\n", - "\tif (WARN_ON_ONCE(ret)) {\n", - "\t\tpr_warn(\"error enabling all events\\n\");\n", - "\t\treturn;\n", - "\t}\n", - "\n", - "\tcancel_delayed_work_sync(&req->work);\n", - "\n", - "\ttrace_pm_qos_update_request(req->pm_qos_class,\n", - "\t\t\t\t\t new_value, timeout_us);\n", - "\tif (new_value != req->node.prio)\n", - "\t\tpm_qos_update_target(struct pm_qos_constraints network_tput_constraints,\n", - "\t.name = \"memory_bandwidth\",\n", - "};\n", - "\n", - "\n", - "static struct pm_qos_object memory_bandwidth_pm_qos,\n", - "};\n", - "\n", - "static ssize_t\n", - "tracing_write_stub(struct file *filp, const char __user *ubuf, size_t cnt,\n", - "\t\t loff_t *ppos)\n", - "{\n", - "\tint ret = -ENODEV;\n", - "\n", - "\tmutex_lock(&trace_types_lock);\n", - "\n", - "\ttr->current_trace = &nop_trace;\n", - "\n", - "#ifdef CONFIG_BRANCH_TRACER\n", - "int\n", - "trace_selftest_startup_wakeup,\n", - "#endif\n", - "\t.open\t\t= wakeup_trace_open,\n", - "\t.close\t\t= graph_trace_close,\n", - "\t.init\t\t= graph_trace_init,\n", - "\t.reset\t\t= function_trace_start,\n", - "\t.flags\t\t= &func_flags,\n", - "\t.set_flag\t= wakeup_set_flag,\n", - "\t.flag_changed\t= irqsoff_flag_changed,\n", - "#ifdef CONFIG_FTRACE_SELFTEST\n", - "\t.selftest = trace_selftest_startup_preemptirqsoff(struct tracer *tracer = tr->current_trace;\n", - "\tinfo->iter.trace_buffer = &tr->trace_buffer;\n", - "\tinfo->spare\t\t= NULL;\n", - "\t/* Force reading ring buffer for first read */\n", - "\tinfo->read\t\t= (unsigned int)-1;\n", - "\n", - "\tfilp->private_data = dir;\n", - "\n", - "\treturn 0;\n", - "}\n", - "\n", - "static void mmio_trace_start(struct trace_iterator *iter, int flags,\n", - "\t\t struct timespec *tp)\n", - "{\n", - "\treturn posix_cpu_clock_get,\n", - "\t.timer_create\t= posix_cpu_timer_create(&timer);\n", - "\ttimer.it_process = current;\n", - "\tif (!error) {\n", - "\t\tstatic struct itimerspec zero_it;\n", - "\n", - "\t\tmemset(it, 0, sizeof *it);\n", - "\t\tit->it_value = *rqtp;\n", - "\n", - "\t\tspin_lock_irq(&callback_lock);\n", - "\n", - "\tif (!cpumask_empty(desc->percpu_enabled));\n", - "\t\tgoto bad;\n", - "\t}\n", - "\n", - "\t/* Found it - now remove it from the stack, and add back the other\n", - "\t * entries (if any), recalculating the hash along the way:\n", - "\t */\n", - "\n", - "\tcurr->lockdep_depth++;\n", - "\tcheck_chain_key(curr);\n", - "}\n", - "\n", - "static int __lock_is_held(struct lockdep_map *lock)\n", - "{\n", - "\tstruct task_struct *p, int head)\n", - "{\n", - "\tstruct sched_rt_entity *rt_se)\n", - "{\n", - "\treturn !list_empty(&pool->worklist) &&\n", - "\t\tatomic_read(&pool->nr_running);\n", - "\t}\n", - "}\n", - "\n", - "static void update_cond_flag(struct ftrace_event_field *field;\n", - "\tstruct list_head *head_page, *prev_page, *r;\n", - "\t\tstruct list_head *firing,\n", - "\t\t unsigned long len,\n", - "\t\t\t unsigned long len,\n", - "\t\t\t\t struct load_info *info)\n", - "{\n", - "}\n", - "\n", - "static void add_kallsyms(struct module *mod)\n", - "{\n", - "\tdel_usage_links(mod);\n", - "\tadd_sect_attrs(mod, info);\n", - "\tadd_notes_attrs(mod, info);\n", - "\n", - "\tkobject_uevent(&mod->mkobj.kobj, 0, sizeof(mod->mkobj.kobj));\n", - "\tmod->mkobj.kobj.kset = module_kset;\n", - "\t\terr = kobject_init_and_add(&mk->kobj, &module_uevent.attr);\n", - "#endif\n", - "\t\tif (err) {\n", - "\t\t\tkobject_put(&mk->kobj);\n", - "\t\t}\n", - "\t}\n", - "}\n", - "\n", - "/* module\n" - ] - } - ], - "source": [ - "gen(train_LM(linux, order=20), length=3000)" + "LanguageModel(20, linux).gen(length=3000)" ] }, { @@ -1094,31 +1046,27 @@ "\n", "As Goldberg says, \"Order 10 is pretty much junk.\" But order 20 is much better. Most of the comments have a start and an end; most of the open parentheses are balanced with a close parentheses; but the braces are not as well balanced. That shouldn't be surprising. If the span of an open/close parenthesis pair is less than 20 characters then it can be represented within the model, but if the span of an open/close brace is more than 20 characters, then it cannot be represented by the model. Goldberg notes that Karpathy's RRN seems to have learned to devote some of its long short-term memory (LSTM) to representing nesting level, as well as things like whether we are currently within a string or a comment. It is indeed impressive, as Karpathy says, that the model learned to do this on its own, without any input from the human engineer.\n", "\n", - "## Token Models versus Character Models\n", + "## Character Models versus Word and Token Models\n", "\n", "Karpathy and Goldberg both used character models, because the exact formatting of characters (especially indentation and line breaks) is important in the format of plays and C++ programs. But if you are interested in generating paragraphs of text that don't have any specific format, it is common to use a **word** model, which represents the probability of the next word given the previous words, or a **token** model in which tokens can be words, punctuation, or parts of words. For example, the text `\"Spiderman!\"` might be broken up into the three tokens `\"Spider\"`, `\"man\"`, and `\"!\"`. \n", "\n", - "One simple way of tokenizing a text is to break it up into alternating strings of word and non-word characters; the function `tokenize` does that by default:" + "One simple way of tokenizing a text is to break it up into alternating strings of word and non-word characters; the function `tokenize` does that:" ] }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "import re\n", "\n", - "word_or_nonword = r'\\w+|\\W+' # Regular expression to parse a string of either word or non-word characters.\n", - "\n", - "def tokenize(text: str, regex=word_or_nonword) -> List[Token]: \n", - " \"\"\"Break text up into tokens using regex.\"\"\"\n", - " return re.findall(regex, text)" + "tokenize = re.compile(r'\\w+|\\W+').findall # Find all alternating word- or non-word strings" ] }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 24, "metadata": {}, "outputs": [], "source": [ @@ -1133,22 +1081,22 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can train a token model on the Shakespeare data. A model of order 6 keeps a history of up to three word and three non-word tokens. " + "We can train a token language model on the Shakespeare data. A model of order 6 keeps a history of up to three word and three non-word tokens. " + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [], + "source": [ + "TLM = LanguageModel(6, tokenize(shakespeare))" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, - "outputs": [], - "source": [ - "TLM = train_LM(tokenize(data), order=6)" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": {}, "outputs": [ { "data": { @@ -1156,7 +1104,7 @@ "Counter({'Romeo': 1})" ] }, - "execution_count": 27, + "execution_count": 26, "metadata": {}, "output_type": "execute_result" } @@ -1167,7 +1115,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 27, "metadata": {}, "outputs": [ { @@ -1176,7 +1124,7 @@ "Counter({'stars': 1, 'Grecian': 1})" ] }, - "execution_count": 28, + "execution_count": 27, "metadata": {}, "output_type": "execute_result" } @@ -1187,7 +1135,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 28, "metadata": {}, "outputs": [ { @@ -1196,7 +1144,7 @@ "Counter({'life': 1, 'business': 1, 'dinner': 1, 'time': 1})" ] }, - "execution_count": 29, + "execution_count": 28, "metadata": {}, "output_type": "execute_result" } @@ -1207,7 +1155,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 29, "metadata": {}, "outputs": [ { @@ -1216,7 +1164,7 @@ "Counter({' ': 2})" ] }, - "execution_count": 30, + "execution_count": 29, "metadata": {}, "output_type": "execute_result" } @@ -1234,7 +1182,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 30, "metadata": {}, "outputs": [ { @@ -1242,97 +1190,63 @@ "output_type": "stream", "text": [ "First Citizen:\n", - "Before we proceed any further, hear me speak,\n", - "Before you answer Warwick. His demand\n", - "Springs not from Edward's well-meant honest love,\n", - "But from deceit bred by necessity;\n", - "For how can I grace my talk,\n", - "Wanting a hand to hold a sceptre up\n", - "And with the clamour of thy drum,\n", - "And even at hand a drum is ready braced\n", - "That shall reverberate all as loud as Mars. By Jupiter,\n", - "Were I the Moor, I would not be awaked.\n", + "Before we proceed any further, hear me speak.\n", "\n", - "LORENZO:\n", - "That is the very note of it: and it is known she is, these moral laws\n", - "Of nature and of nations, 'long\n", - "To him and his virtue;\n", - "By her election may be truly read\n", - "What kind of man,\n", - "So keen and greedy to confound a man:\n", - "He plies the duke at dinner: by two o'clock I'll get me such a question: stand again:\n", - "Think'st thou I am an old man's life is done:\n", - "Then, dear my liege, mine honour let me try;\n", - "In that I live and for that will I cause these of Cyprus to\n", - "mutiny; whose qualification shall come into no true\n", - "taste again but by the recorder.\n", - "Then he was urged to tell my tale again,\n", - "'Thus saith the duke, thus hath the duke inferr'd;'\n", - "But nothing spake in warrant from himself.\n", - "When he had done, some followers of mine own lie heavy in my breast\n", - "And go well satisfied to France again.\n", + "TIMON:\n", + "Freely, good father.\n", "\n", - "PRINCESS:\n", - "You do the nobler.\n", + "Old Athenian:\n", + "Thou hast a sister by the mother's, from the top to toe?\n", "\n", - "CORIOLANUS:\n", - "I muse my Lord of Winchester, I charge you,\n", - "Not fearing the displeasure of your master,\n", - "Which on your just proceeding I'll keep what I have said to you?\n", + "MARCELLUS:\n", + "My lord, upon the platform where we watch'd.\n", "\n", - "OPHELIA:\n", - "So please you, it is true, indeed.\n", + "HAMLET:\n", + "Did you not tell me, Griffith, as thou led'st me,\n", + "That the great body of our kingdom\n", + "How foul it is; what rank diseases grow\n", + "And with what zeal! for, now he has crack'd the league, and hath attach'd\n", + "Our merchants' goods at Bourdeaux.\n", "\n", - "GRATIANO:\n", - "'Tis a strange serpent.\n", + "ABERGAVENNY:\n", + "Is it therefore\n", + "The ambassador is silenced?\n", "\n", - "MARK ANTONY:\n", - "'Tis said, man; and farewell.\n", + "NORFOLK:\n", + "Marry, is't.\n", "\n", - "EROS:\n", - "Farewell, great chief. Shall I strike at it with you, and so following, but I will murder your ruff for this.\n", + "ABERGAVENNY:\n", + "A proper title of a peace; and purchased\n", + "At a superfluous rate!\n", "\n", - "FALSTAFF:\n", - "No more, Pistol; I would not\n", - "Believe her lips in opening it. Proceed.\n", + "BUCKINGHAM:\n", + "Why, all this business\n", + "Our reverend cardinal carried.\n", "\n", - "CORNELIUS:\n", - "Your daughter, whom she bore in hand to love\n", - "With such integrity, she did confess\n", - "Was as a scorpion to her sight; whose life,\n", - "But that the heavens fought: the king himself\n", - "Of his wings destitute, the army broken,\n", - "And but the backs of Britons seen, all flying\n", - "Through a straight lane; the enemy full-hearted,\n", - "Lolling the tongue with slaughtering, having work\n", - "More plentiful than tools to do't?\n", + "NORFOLK:\n", + "Like it your grace,\n", + "The Breton navy is dispersed by tempest:\n", + "Richmond, in Yorkshire, sent out a boat\n", + "Unto the shore, to ask those on the banks\n", + "If they were known, as the suspect is great,\n", + "Would make thee quickly hop without thy head.\n", + "Give me my horse, you\n", + "rogues; give me my gown; or else keep it in your age.\n", + "Then, in the name of something holy, sir, why stand you\n", + "In this strange stare?\n", "\n", - "First Lord:\n", - "You must not think\n", - "That we are made of stuff so flat and dull\n", - "That we can let our beard be shook with danger\n", - "And think it not the worst of me. So, I leave you, sir.\n", - "\n", - "VINCENTIO:\n", - "You shall not now be stol'n away to Rome; hath ta'en thy stand,\n", - "The elected deer before thee?\n", - "\n", - "PISANIO:\n", - "But to win time\n", - "To lose so bad employment; in the which\n", - "I do arrest you, sir: you might have heard it said, unbidden guests\n", - "Are often welcomest when they are full,\n", - "They \n" + "ALONSO:\n", + "O, \n" ] } ], "source": [ - "gen(TLM)" + "TLM.gen(400)" ] }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 31, "metadata": {}, "outputs": [ { @@ -1346,95 +1260,37 @@ "Peace, ho! Hear Antony. Most noble Antony!\n", "\n", "ANTONY:\n", - "Why, friends, you go to do you know not what\n", - "you speak. But, if ever the duke return, as our\n", - "prayers are he may, let me desire you to make your\n", - "answer before him. If it be honest you have spoke,\n", - "you have courage to maintain it: I am bound to wonder, I am bound\n", - "To Persia, and want guilders for my voyage:\n", - "Therefore make present satisfaction,\n", - "Or I'll attach you by this officer.\n", + "Why, friends, you go to do you know not what:\n", + "Wherein hath Caesar thus deserved your loves?\n", + "Alas, you know not: I must tell you that,\n", + "Before my daughter told me--what might you,\n", + "Or my dear majesty your queen here, think,\n", + "If I had play'd the desk or table-book,\n", + "Or given my heart a winking, mute and dumb,\n", + "Or look'd upon this love with idle sight;\n", + "What might you think? No, I went round to work,\n", + "And my young mistress thus I did bespeak:\n", + "'Lord Hamlet is a prince, out of thy star;\n", + "This must not be:' and then I precepts gave her,\n", + "That she should lock herself from his resort,\n", + "Admit no messengers, receive no tokens.\n", + "Which done, she took the fruits of my advice;\n", + "And he, repulsed--a short tale to make--\n", + "Fell into a sadness, then into a fast,\n", + "Thence to a watch, thence into a weakness,\n", + "Thence to a lightness, and, by this declension,\n", + "Into the madness wherein now he raves,\n", + "And all we mourn for.\n", "\n", - "ANTIPHOLUS OF EPHESUS:\n", - "I will debate this matter at more leisure\n", - "And teach your ears to list me with more heed.\n", - "To Adriana, villain, hie thee straight:\n", - "Give her this key, and tell her, in the desk\n", - "That's cover'd o'er with Turkish tapestry,\n", - "There is a purse of ducats; let her send it:\n", - "Tell her I am arrested in the street\n", - "And that shall bail me; hie thee, slave, be gone!\n", - "On, officer, to prison till it come.\n", + "KING CLAUDIUS:\n", + "Do you think 'tis this?\n", "\n", - "DROMIO OF SYRACUSE:\n", - "To Adriana! that is where we dined,\n", - "Where Dowsabel did claim me for her husband:\n", - "She is too big, I hope, for me to compass.\n", - "Thither I must, although against my will,\n", - "For servants must their masters' minds fulfil.\n", - "\n", - "ADRIANA:\n", - "Ah, Luciana, did he tempt thee so?\n", - "Mightst thou perceive austerely in his eye\n", - "That he did plead in earnest? yea or no?\n", - "Look'd he or red or pale, or sad or merrily,\n", - "Interpretation will misquote our looks,\n", - "And we shall feed like oxen at a stall,\n", - "The better cherish'd, still the nearer death.\n", - "My nephew's trespass may be well forgot;\n", - "it hath the excuse of youth and heat of blood,\n", - "And an adopted name of privilege,\n", - "A hair-brain'd Hotspur, govern'd by a child!\n", - "\n", - "Second Citizen:\n", - "In him there is a hope of government,\n", - "That in his nonage council under him,\n", - "And in his full and ripen'd years himself,\n", - "No doubt, shall then and till then govern well.\n", - "\n", - "First Citizen:\n", - "So stood the state when Henry the Sixth\n", - "Was crown'd in Paris but at nine months old.\n", - "\n", - "Third Citizen:\n", - "Stood the state so? No, no, good friends, God wot;\n", - "For then this land was famously enrich'd\n", - "With politic grave counsel; then the king\n", - "Had virtuous uncles to protect his grace.\n", - "\n", - "First Citizen:\n", - "Why, so hath this, both by the father and mother.\n", - "\n", - "Third Citizen:\n", - "Better it were they all came by the father,\n", - "Or by the father there were none at all;\n", - "For emulation now, who shall be nearest,\n", - "Will touch us all too near, if God prevent not.\n", - "O, full of danger is the Duke of Norfolk:\n", - "Ratcliff, thyself, or Catesby; where is he?\n", - "\n", - "CATESBY:\n", - "Here, my lord.\n", - "\n", - "KING RICHARD III:\n", - "Then, by myself--\n", - "\n", - "QUEEN ELIZABETH:\n", - "Thyself thyself misusest.\n", - "\n", - "KING RICHARD III:\n", - "Why, Buckingham, I say, I would be king,\n", - "\n", - "BUCKINGHAM:\n", - "Why, so you are, my thrice renowned liege.\n", - "\n", - "KING RICHARD III:\n", "\n" ] } ], "source": [ - "gen(train_LM(tokenize(data), 8))" + "LanguageModel(8, tokenize(shakespeare)).gen(400)" ] }, { @@ -1448,7 +1304,7 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 32, "metadata": {}, "outputs": [ { @@ -1460,496 +1316,199 @@ " *\n", " * Copyright (C) 1992, 1998-2006 Linus Torvalds, Ingo Molnar\n", " *\n", - " * This file contains the IRQ-resend code\n", - " *\n", - " * If the interrupt is waiting to be processed, we try to re-run it.\n", - " * We can't directly run it from here since the caller might be in an\n", - " * interrupt-protected region. Not all irq controller chips can\n", - " * retrigger interrupts at the hardware level, so in those cases\n", - " * we allow the resending of IRQs via a tasklet.\n", + " * This file contains the private data structure and API definitions.\n", " */\n", "\n", - "#include \n", - "#include \n", - "#include \n", - "#include \n", - "#include \n", - "#include \n", - "#include \n", - "#include \n", - "#include \n", + "#ifndef __KERNEL_RTMUTEX_COMMON_H\n", + "#define __KERNEL_RTMUTEX_COMMON_H\n", "\n", - "#include \"trace.h\"\n", - "\n", - "static DEFINE_PER_CPU(int, bpf_prog_active);\n", - "\n", - "/**\n", - " * trace_call_bpf - invoke BPF program\n", - " * @prog: BPF program\n", - " * @ctx: opaque context pointer\n", - " *\n", - " * kprobe handlers execute BPF programs via this helper.\n", - " * Can be used from static tracepoints in the future.\n", - " *\n", - " * Return: BPF programs always return an integer which is interpreted by\n", - " * kprobe handler as:\n", - " * 0 - return from kprobe (event is filtered out)\n", - " * 1 - store kprobe event into ring-buffer,\n", - "\t\t * so return zero here\n", - "\t\t */\n", - "\t\tret = 0;\n", - "\t\tgoto out;\n", - "\t}\n", - "\n", - "\tif (trigger) {\n", - "\t\tnumber = strsep(&trigger, \":\");\n", - "\n", - "\t\tret = -EINVAL;\n", - "\t\tif (!strlen(number))\n", - "\t\t\tgoto out_free;\n", - "\n", - "\t\t/*\n", - "\t\t * We use the callback data field (which is a pointer)\n", - "\t\t * as our counter.\n", - "\t\t */\n", - "\t\tret = kstrtoul(number, 0, &data->count);\n", - "\tif (ret)\n", - "\t\tgoto out_free;\n", - "\n", - " out_reg:\n", - "\t/* Don't let event modules unload while probe registered */\n", - "\tret = try_module_get(file->event_call->mod);\n", - "\tif (!ret) {\n", - "\t\tret = -EBUSY;\n", - "\t\tgoto out_free;\n", - "\t}\n", - "\n", - "\tret = __ftrace_event_enable_disable(file, 1, 1);\n", - "\tif (ret < 0)\n", - "\t\tgoto skip_full_check;\n", - "\n", - "\tenv->explored_states = kcalloc(env->prog->len,\n", - "\t\t\t\t sizeof(struct verifier_state_list *),\n", - "\t\t\t\t GFP_USER);\n", - "\tret = -ENOMEM;\n", - "\tif (!env->explored_states)\n", - "\t\tgoto skip_full_check;\n", - "\n", - "\tret = check_cfg(env);\n", - "\tif (ret < 0)\n", - "\t\tgoto out_put;\n", - "\tret = cmd_ops->reg(glob, trigger_ops, trigger_data, file);\n", - "\t/*\n", - "\t * The above returns on success the # of functions enabled,\n", - "\t * but if it didn't find any functions it returns zero.\n", - "\t * Consider no functions a failure too.\n", - "\t */\n", - "\tif (!ret) {\n", - "\t\tret = -ENOENT;\n", - "\t\tgoto out_disable;\n", - "\t} else if (ret < 0)\n", - "\t\tgoto out_free;\n", - "\n", - " out_reg:\n", - "\t/* Don't let event modules unload while probe registered */\n", - "\tret = try_module_get(file->event_call->mod);\n", - "\tif (!ret) {\n", - "\t\tret = -EBUSY;\n", - "\t\tgoto out_free;\n", - "\t}\n", - "\n", - "\tret = trace_event_enable_disable(event_enable_file, 1, 1);\n", - "\tif (ret < 0)\n", - "\t\tgoto out_disable;\n", - "\t/* Just return zero, not the number of enabled functions */\n", - "\tret = 0;\n", - " out:\n", - "\tmutex_unlock(&ftrace_lock);\n", - "\n", - "\treturn ret;\n", - "}\n", - "\n", - "#ifdef CONFIG_MODULES\n", - "\n", - "#define next_to_ftrace_page(p) container_of(p, struct ftrace_page, next)\n", - "\n", - "void ftrace_release_mod(struct module *mod)\n", - "{\n", - "\tstruct module_attribute *attr;\n", - "\tint i;\n", - "\n", - "\tfor (i = KDB_MAXBPT - 1; i >= 0; i--)\n", - "\t\tif (rdp->nxttail[i] == rdp->nxttail[RCU_DONE_TAIL])\n", - "\t\t\t\trdp->nxttail[i] = rsp->orphan_donetail;\n", - "\t\trsp->orphan_donelist = NULL;\n", - "\t\trsp->orphan_donetail = &rsp->orphan_donelist;\n", - "\t}\n", - "\tif (rsp->orphan_nxtlist != NULL) {\n", - "\t\t__call_rcu_nocb_enqueue(rdp, rsp->orphan_nxtlist,\n", - "\t\t\t\t\trsp->orphan_nxttail, ql, qll, flags);\n", - "\t\tql = qll = 0;\n", - "\t\trsp->orphan_nxtlist = NULL;\n", - "\t\trsp->orphan_nxttail = &rsp->orphan_nxtlist;\n", - "\t}\n", - "\treturn true;\n", - "}\n", + "#include \n", "\n", "/*\n", - " * If necessary, kick off a new grace period if one is needed.\n", - "\t */\n", - "\trcu_report_qs_rsp(rsp, flags); /* releases rnp->lock. */\n", - "}\n", - "\n", - "/*\n", - " * Record a quiescent state for all tasks that were preempted within an RCU read-side critical\n", - " * sections. This function also enables RCU lockdep checking.\n", + " * The rtmutex in kernel tester is independent of rtmutex debugging. We\n", + " * call schedule_rt_mutex_test() instead of schedule() for the tasks which\n", + " * belong to the tester. That way we can delay the wakeup path of those\n", + " * threads to provoke lock stealing and testing of complex boosting scenarios.\n", " */\n", - "void rcu_scheduler_starting(void)\n", + "#ifdef CONFIG_RT_MUTEX_TESTER\n", + "\n", + "extern void schedule_rt_mutex_test(struct rt_mutex *mutex)\n", "{\n", - "\tWARN_ON(num_online_cpus() != 1);\n", - "\tWARN_ON(nr_context_switches() > 0);\n", - "\trcu_scheduler_active = 1;\n", - "}\n", + "\tint tid, op, dat;\n", + "\tstruct test_thread_data *td;\n", "\n", - "/*\n", - " * Compute the per-level fanout, either using the exact fanout specified\n", - " * or balancing the tree, depending on CONFIG_RCU_FANOUT_EXACT.\n", - " */\n", - "static void __init rcu_init_levelspread(struct rcu_state *rsp)\n", - "{\n", - "\tint cpu;\n", - "\tunsigned long sum = 0;\n", - "\tunsigned long t;\n", - "\n", - "\tfor_each_possible_cpu(cpu) {\n", - "\t\tt = ACCESS_ONCE(per_cpu_ptr(sp->per_cpu_ref, cpu)->c[1]);\n", - "\t}\n", - "\treturn sum;\n", - "}\n", - "\n", - "/**\n", - " * cleanup_srcu_struct - deconstruct a sleep-RCU structure\n", - " * @sp: structure to clean up.\n", - " *\n", - " * Must invoke this after you are finished using a given srcu_struct that\n", - " * was initialized via init_srcu_struct(), else you leak memory.\n", - " */\n", - "void cleanup_srcu_struct(struct srcu_struct *sp)\n", - "{\n", - "\tif (WARN_ON(srcu_readers_active(sp)))\n", - "\t\treturn; /* Leakage unless caller handles error. */\n", - "\tfree_percpu(sp->per_cpu_ref);\n", - "\tsp->per_cpu_ref = NULL;\n", - "}\n", - "EXPORT_SYMBOL_GPL(cleanup_srcu_struct);\n", - "\n", - "/*\n", - " * Counts the new reader in the appropriate per-CPU\n", - " * element of the srcu_struct. Note that this may well be a different\n", - " * CPU than that which was incremented by the corresponding srcu_read_lock().\n", - " * Must be called from process context.\n", - " *\n", - " * Note that it is permissible to omit this call entirely, as is\n", - " * done in architectures that do no CPU-hotplug error checking.\n", - " */\n", - "int cpu_check_up_prepare(int cpu)\n", - "{\n", - "\tif (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) {\n", - "\t\tatomic_set(&per_cpu(cpu_hotplug_state, cpu), CPU_UP_PREPARE);\n", - "\t\treturn 0;\n", - "\t}\n", - "\n", - "\tswitch (atomic_read(&per_cpu(cpu_hotplug_state, cpu)) == CPU_DEAD)\n", - "\t\tgoto update_state;\n", - "\tudelay(5);\n", - "\n", - "\t/* But if the outgoing CPU dawdles, wait increasingly long times. */\n", - "\twhile (atomic_read(&per_cpu(cpu_hotplug_state, cpu)) != CPU_DEAD) {\n", - "\t\tschedule_timeout_uninterruptible(sleep_jf);\n", - "\t\tjf_left -= sleep_jf;\n", - "\t\tif (jf_left <= 0)\n", + "\t/* We have to lookup the task */\n", + "\tfor (tid = 0; tid < MAX_RT_TEST_THREADS; tid++) {\n", + "\t\tif (threads[tid] == current)\n", "\t\t\tbreak;\n", - "\t\tsleep_jf = DIV_ROUND_UP(sleep_jf * 11, 10);\n", "\t}\n", - "update_state:\n", - "\toldstate = atomic_read(&per_cpu(cpu_hotplug_state, cpu));\n", - "\t\tif (oldstate != CPU_BROKEN)\n", - "\t\t\tnewstate = CPU_DEAD;\n", - "\t\telse\n", - "\t\t\tnewstate = CPU_DEAD_FROZEN;\n", - "\t} while (atomic_cmpxchg(&per_cpu(cpu_hotplug_state, cpu),\n", - "\t\t\t\t oldstate, CPU_BROKEN) != oldstate)\n", - "\t\t\tgoto update_state;\n", - "\t\tret = false;\n", + "\n", + "\tBUG_ON(tid == MAX_RT_TEST_THREADS);\n", + "\n", + "\ttd = &thread_data[tid];\n", + "\n", + "\top = td->opcode;\n", + "\tdat = td->opdata;\n", + "\n", + "\tswitch (op) {\n", + "\tcase RTTEST_LOCK:\n", + "\tcase RTTEST_LOCKINT:\n", + "\tcase RTTEST_LOCKNOWAIT:\n", + "\tcase RTTEST_LOCKINTNOWAIT:\n", + "\t\tif (mutex != &mutexes[dat])\n", + "\t\t\tbreak;\n", + "\n", + "\t\tif (td->mutexes[dat] != 2)\n", + "\t\t\treturn;\n", + "\n", + "\t\ttd->mutexes[dat] = 3;\n", + "\t\ttd->event = atomic_add_return(1, &rttest_event);\n", + "\t\tbreak;\n", + "\n", + "\tcase RTTEST_LOCKNOWAIT:\n", + "\tcase RTTEST_LOCKINTNOWAIT:\n", + "\t\tif (mutex != &mutexes[dat])\n", + "\t\t\tbreak;\n", + "\n", + "\t\tif (td->mutexes[dat] != 1)\n", + "\t\t\tbreak;\n", + "\n", + "\t\ttd->mutexes[dat] = 2;\n", + "\t\ttd->event = atomic_add_return(1, &rttest_event);\n", + "\t\tbreak;\n", + "\n", + "\tdefault:\n", + "\t\tbreak;\n", "\t}\n", - "\treturn ret;\n", + "\n", + "\tschedule();\n", + "\n", + "\n", + "\tswitch (op) {\n", + "\tcase RTTEST_LOCK:\n", + "\tcase RTTEST_LOCKINT:\n", + "\t\tif (mutex != &mutexes[dat])\n", + "\t\t\treturn;\n", + "\n", + "\t\tif (td->mutexes[dat] != 2)\n", + "\t\t\treturn;\n", + "\n", + "\t\ttd->mutexes[dat] = 1;\n", + "\t\ttd->event = atomic_add_return(1, &rttest_event);\n", + "\t\trt_mutex_lock(&mutexes[id]);\n", + "\t\ttd->event = atomic_add_return(1, &rttest_event);\n", + "\t\tbreak;\n", + "\n", + "\tcase RTTEST_LOCKNOWAIT:\n", + "\tcase RTTEST_LOCKINTNOWAIT:\n", + "\t\tif (mutex != &mutexes[dat])\n", + "\t\t\treturn;\n", + "\n", + "\t\tif (td->mutexes[dat] != 2)\n", + "\t\t\treturn;\n", + "\n", + "\t\ttd->mutexes[dat] = 3;\n", + "\t\ttd->event = atomic_add_return(1, &rttest_event);\n", + "\t\trt_mutex_unlock(&mutexes[id]);\n", + "\t\ttd->event = atomic_add_return(1, &rttest_event);\n", + "\t\treturn 0;\n", + "\n", + "\tcase RTTEST_RESET:\n", + "\t\tfor (i = 0; i < AUDIT_BITMASK_SIZE; i++)\n", + "\t\tentry->rule.mask[i] = rule->mask[i];\n", + "\n", + "\tfor (i = 0; i < ALARM_NUMTYPE; i++) {\n", + "\t\tstruct alarm_base *base = &alarm_bases[alarm->type];\n", + "\tunsigned long flags;\n", + "\n", + "\tspin_lock_irqsave(&callback_lock, flags);\n", + "\trcu_read_lock();\n", + "\tguarantee_online_cpus(task_cs(tsk), pmask);\n", + "\trcu_read_unlock();\n", + "\tspin_unlock_irqrestore(&callback_lock, flags);\n", "}\n", "\n", - "/*\n", - " * Called by the outgoing CPU to report its successful death. Return\n", - " * false if this report follows the surviving CPU's timing out.\n", - " *\n", - " * A separate \"CPU_DEAD_FROZEN\" is used when the surviving CPU\n", - " * timed out. This approach allows architectures to omit calls to\n", - " * cpu_check_up_prepare() and cpu_set_state_online() without defeating\n", - " * the next cpu_wait_death()'s polling loop.\n", - " */\n", - "bool cpu_report_death(void)\n", + "void cpuset_cpus_allowed_fallback(struct task_struct *tsk)\n", "{\n", - "\tint oldstate;\n", - "\tint newstate;\n", - "\tint cpu = smp_processor_id();\n", + "\trcu_read_lock();\n", + "\tdo_set_cpus_allowed(tsk, task_cs(tsk)->effective_cpus);\n", + "\trcu_read_unlock();\n", "\n", - "\tdo {\n", - "\t\toldstate = atomic_read(&per_cpu(cpu_hotplug_state, cpu));\n", - "\tif (oldstate == CPU_DEAD) {\n", - "\t\t/* Outgoing CPU died normally, update state. */\n", - "\t\tsmp_mb(); /* atomic_read() before update. */\n", - "\t\tatomic_set(&per_cpu(cpu_hotplug_state, cpu), CPU_POST_DEAD);\n", - "\t} else {\n", - "\t\t/* Outgoing CPU still hasn't gotten around\n", - " * to dying. In the latter two cases, the CPU might not be set up\n", - " * properly, but it is up to the arch-specific code:\n", + "\t/*\n", + "\t * We own tsk->cpus_allowed, nobody can change it under us.\n", "\t *\n", - "\t * \tinsn -\tcopy_insn() saves the original instruction here for\n", - "\t *\t\tarch_uprobe_analyze_insn().\n", + "\t * But we used cs && cs->cpus_allowed lockless and thus can\n", + "\t * race with cgroup_attach_task() or update_cpumask() and get\n", + "\t * the wrong tsk->cpus_allowed. However, both cases imply the\n", + "\t * subsequent cpuset_change_cpumask()->set_cpus_allowed_ptr()\n", + "\t * which takes task_rq_lock().\n", "\t *\n", - "\t *\tixol -\tpotentially modified instruction to execute out of\n", - "\t *\t\tline, copied to xol_area by xol_get_insn_slot().\n", + "\t * If we are called after it dropped the lock we must see all\n", + "\t * changes in tsk_cs()->cpus_allowed. Otherwise we can temporary\n", + "\t * set any mask even if it is interleaved with any other text.\n", "\t */\n", - "\tstruct arch_uprobe\tarch;\n", - "};\n", - "\n", - "struct return_instance {\n", - "\tstruct uprobe\t\t*uprobe;\n", - "\tunsigned long\t\tfunc;\n", - "\tunsigned long\t\tret_ip;\n", - "};\n", - "\n", - "/*\n", - " * trace_flag_type is an enumeration that defines bit\n", - " * positions into trace_flags that controls the output.\n", - " *\n", - " * NOTE: These bits must match the trace_options array in\n", - " * trace.c.\n", - " */\n", - "enum trace_iterator_flags {\n", - "\tTRACE_ITER_PRINT_PARENT\t\t= 0x01,\n", - "\tTRACE_ITER_SYM_OFFSET\t\t= 0x02,\n", - "\tTRACE_ITER_SYM_ADDR\t\t= 0x04,\n", - "\tTRACE_ITER_VERBOSE\t\t= 0x08,\n", - "\tTRACE_ITER_RAW\t\t\t= 0x10,\n", - "\tTRACE_ITER_HEX\t\t\t= 0x20,\n", - "\tTRACE_ITER_BIN\t\t\t= 0x40,\n", - "\tTRACE_ITER_BLOCK\t\t= 0x80,\n", - "\tTRACE_ITER_STACKTRACE\t\t= 0x100,\n", - "\tTRACE_ITER_PRINTK\t\t= 0x200,\n", - "\tTRACE_ITER_PREEMPTONLY\t\t= 0x400,\n", - "\tTRACE_ITER_BRANCH\t\t= 0x800,\n", - "\tTRACE_ITER_ANNOTATE\t\t= 0x1000,\n", - "\tTRACE_ITER_USERSTACKTRACE = 0x2000,\n", - "\tTRACE_ITER_SYM_USEROBJ = 0x4000,\n", - "\tTRACE_ITER_PRINTK_MSGONLY\t= 0x8000,\n", - "\tTRACE_ITER_CONTEXT_INFO\t\t= 0x10000, /* Print pid/cpu/time */\n", - "\tTRACE_ITER_LATENCY_FMT\t\t= 0x20000,\n", - "\tTRACE_ITER_SLEEP_TIME\t\t= 0x40000,\n", - "\tTRACE_ITER_GRAPH_TIME\t\t= 0x80000,\n", - "\tTRACE_ITER_RECORD_CMD\t\t= 0x100000,\n", - "\tTRACE_ITER_OVERWRITE\t\t= 0x200000,\n", - "\tTRACE_ITER_STOP_ON_FREE\t\t= 0x400000,\n", - "\tTRACE_ITER_IRQ_INFO\t\t= 0x800000,\n", - "\tTRACE_ITER_MARKERS\t\t= 0x1000000,\n", - "\tTRACE_ITER_FUNCTION\t\t= 0x2000000,\n", - "};\n", - "\n", - "/*\n", - " * TRACE_ITER_SYM_MASK masks the options in trace_flags that\n", - " * control the output of kernel symbols.\n", - " */\n", - "#define TRACE_ITER_SYM_MASK \\\n", - "\t(TRACE_ITER_PRINT_PARENT|TRACE_ITER_SYM_OFFSET|TRACE_ITER_SYM_ADDR)\n", - "\n", - "extern struct tracer nop_trace;\n", - "\n", - "#ifdef CONFIG_BRANCH_TRACER\n", - "extern int enable_branch_tracing(struct trace_array *tr)\n", - "{\n", - "\tmutex_lock(&event_mutex);\n", - "\n", - "\t/* Disable any event triggers and associated soft-disabled events */\n", - "\tclear_event_triggers(tr);\n", - "\n", - "\t/* Disable any running events */\n", - "\t__ftrace_set_clr_event_nolock(tr, NULL, NULL, NULL, 0);\n", - "\tif (WARN_ON_ONCE(ret)) {\n", - "\t\tpr_warn(\"error on probing function return.\\n\");\n", - "\t\twarn++;\n", + "\tif (!KDB_STATE(PRINTF_LOCK)) {\n", + "\t\tKDB_STATE_SET(PRINTF_LOCK);\n", + "\t\tspin_lock_irqsave(&kdb_printf_lock, flags);\n", + "\t\tgot_printf_lock = 1;\n", + "\t\tatomic_inc(&kdb_event);\n", "\t} else {\n", - "\t\t/* Enable trace point */\n", - "\t\ttk = find_trace_kprobe(\"testprobe\", KPROBE_EVENT_SYSTEM);\n", - "\t\tif (WARN_ON_ONCE(tk == NULL)) {\n", - "\t\t\tpr_warn(\"error on getting probe file.\\n\");\n", - "\t\t\t\twarn++;\n", - "\t\t\t} else\n", - "\t\t\t\tenable_trace_kprobe(tk, file);\n", + "\t\t__acquire(kdb_printf_lock);\n", + "\t}\n", + "\n", + "\tdiag = kdbgetintenv(\"LINES\", &linecount);\n", + "\tif (diag || linecount <= 1)\n", + "\t\tlinecount = 24;\n", + "\n", + "\tdiag = kdbgetintenv(\"COLUMNS\", &colcount);\n", + "\tif (diag || colcount <= 1)\n", + "\t\tcolcount = 80;\n", + "\n", + "\tdiag = kdbgetintenv(\"LOGGING\", &logging);\n", + "\tif (!diag && logging) {\n", + "\t\tconst char *setargs[] = { \"set\", \"LOGGING\", \"0\" };\n", + "\t\tkdb_set(2, setargs);\n", + "\t}\n", + "\n", + "\tkmsg_dump_rewind_nolock(&dumper);\n", + "\twhile (kmsg_dump_get_line_nolock(&dumper, 1, NULL, 0, NULL))\n", + "\t\tn++;\n", + "\n", + "\tif (lines < 0) {\n", + "\t\tif (adjust >= n)\n", + "\t\t\tkdb_printf(\"buffer only contains %d lines, nothing \"\n", + "\t\t\t\t \"printed\\n\", n);\n", + "\t\telse if (adjust - lines >= n)\n", + "\t\t\tkdb_printf(\"buffer only contains %d lines, first \"\n", + "\t\t\t\t \"%d lines printed\\n\", n, lines);\n", "\t\t}\n", - "\t}\n", - "\n", - "\tif (warn)\n", - "\t\tgoto end;\n", - "\n", - "\tret = target(1, 2, 3, 4, 5, 6);\n", - "\n", - "\t/* Disable trace points before removing it */\n", - "\ttk = find_trace_kprobe(\"testprobe\", KPROBE_EVENT_SYSTEM);\n", - "\tif (WARN_ON_ONCE(tk == NULL)) {\n", - "\t\tpr_warn(\"error on getting 2nd new probe.\\n\");\n", - "\t\t\twarn++;\n", - "\t\t} else {\n", - "\t\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\t\tpr_warn(\"error on getting test probe.\\n\");\n", - "\t\twarn++;\n", "\t} else {\n", - "\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\tpr_warn(\"error on getting new probe.\\n\");\n", - "\t\t\twarn++;\n", - "\t\t} else {\n", - "\t\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\t\tpr_warn(\"error on getting probe file.\\n\");\n", - "\t\t\t\twarn++;\n", - "\t\t\t} else\n", - "\t\t\t\tenable_trace_kprobe(tk, file);\n", - "\t\t}\n", + "\t\tlines = n;\n", "\t}\n", "\n", - "\tif (warn)\n", - "\t\tgoto end;\n", + "\tif (skip >= n || skip < 0)\n", + "\t\treturn 0;\n", "\n", - "\tret = target(1, 2, 3, 4, 5, 6);\n", + "\tkmsg_dump_rewind_nolock(&dumper);\n", + "\twhile (kmsg_dump_get_line_nolock(&dumper, 1, NULL, 0, NULL))\n", + "\t\tn++;\n", "\n", - "\t/* Disable trace points before removing it */\n", - "\ttk = find_trace_kprobe(\"testprobe\", KPROBE_EVENT_SYSTEM);\n", - "\tif (WARN_ON_ONCE(tk == NULL)) {\n", - "\t\tpr_warn(\"error on getting 2nd test probe.\\n\");\n", - "\t\twarn++;\n", - "\t} else {\n", - "\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\tpr_warn(\"error on getting probe file.\\n\");\n", - "\t\t\t\twarn++;\n", - "\t\t\t} else\n", - "\t\t\t\tenable_trace_kprobe(tk, file);\n", - "\t\t}\n", - "\t}\n", - "\n", - "\tret = traceprobe_command(\"r:testprobe2 kprobe_trace_selftest_target \"\n", - "\t\t\t\t \"$retval\", create_trace_kprobe);\n", - "\tif (WARN_ON_ONCE(ret)) {\n", - "\t\tpr_warn(\"error on probing function entry.\\n\");\n", - "\t\twarn++;\n", - "\t} else {\n", - "\t\t/* Enable trace point */\n", - "\t\ttk = find_trace_kprobe(\"testprobe\", KPROBE_EVENT_SYSTEM);\n", - "\t\tif (WARN_ON_ONCE(tk == NULL)) {\n", - "\t\t\tpr_warn(\"error on getting 2nd new probe.\\n\");\n", - "\t\t\twarn++;\n", - "\t\t} else {\n", - "\t\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\t\tpr_warn(\"error on getting probe file.\\n\");\n", - "\t\t\t\twarn++;\n", - "\t\t\t} else\n", - "\t\t\t\tenable_trace_kprobe(tk, file);\n", - "\t\t}\n", - "\t}\n", - "\n", - "\tif (warn)\n", - "\t\tgoto end;\n", - "\n", - "\tret = target(1, 2, 3, 4, 5, 6);\n", - "\n", - "\t/* Disable trace points before removing it */\n", - "\ttk = find_trace_kprobe(\"testprobe\", KPROBE_EVENT_SYSTEM);\n", - "\tif (WARN_ON_ONCE(tk == NULL)) {\n", - "\t\tpr_warn(\"error on getting probe file.\\n\");\n", - "\t\t\twarn++;\n", - "\t\t} else\n", - "\t\t\tdisable_trace_kprobe(tk, file);\n", - "\t}\n", - "\n", - "\ttk = find_trace_kprobe(\"testprobe2\", KPROBE_EVENT_SYSTEM);\n", - "\tif (WARN_ON_ONCE(tk == NULL)) {\n", - "\t\tpr_warn(\"error on getting probe file.\\n\");\n", - "\t\t\t\twarn++;\n", - "\t\t\t} else\n", - "\t\t\t\tenable_trace_kprobe(tk, file);\n", - "\t\t}\n", - "\t}\n", - "\n", - "\tif (warn)\n", - "\t\tgoto end;\n", - "\n", - "\tret = target(1, 2, 3, 4, 5, 6);\n", - "\n", - "\t/* Disable trace points before removing it */\n", - "\ttk = find_trace_kprobe(\"testprobe\", KPROBE_EVENT_SYSTEM);\n", - "\tif (WARN_ON_ONCE(tk == NULL)) {\n", - "\t\tpr_warn(\"error on getting 2nd new probe.\\n\");\n", - "\t\t\twarn++;\n", - "\t\t} else {\n", - "\t\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\t\tpr_warn(\"error on getting 2nd test probe.\\n\");\n", - "\t\twarn++;\n", - "\t} else {\n", - "\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\tpr_warn(\"error on getting 2nd new probe.\\n\");\n", - "\t\t\twarn++;\n", - "\t\t} else {\n", - "\t\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\t\tpr_warn(\"error on getting 2nd test probe.\\n\");\n", - "\t\twarn++;\n", - "\t} else {\n", - "\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\tpr_warn(\"error on getting 2nd new probe.\\n\");\n", - "\t\t\twarn++;\n", - "\t\t} else {\n", - "\t\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\t\tpr_warn(\"error on getting probe file.\\n\");\n", - "\t\t\t\twarn++;\n", - "\t\t\t} else\n", - "\t\t\t\tenable_trace_kprobe(tk, file);\n", - "\t\t}\n", - "\t}\n", - "\n", - "\tret = traceprobe_command(\"r:testprobe2 kprobe_trace_selftest_target \"\n", - "\t\t\t\t \"$retval\", create_trace_kprobe);\n", - "\tif (WARN_ON_ONCE(ret)) {\n", - "\t\tpr_warn(\"error on probing function entry.\\n\");\n", - "\t\twarn++;\n", - "\t} else {\n", - "\t\t/* Enable trace point */\n", - "\t\ttk = find_trace_kprobe(\"testprobe2\", KPROBE_EVENT_SYSTEM);\n", - "\t\tif (WARN_ON_ONCE(tk == NULL)) {\n", - "\t\t\tpr_warn(\"error on getting 2nd new probe.\\n\");\n", - "\t\t\twarn++;\n", - "\t\t} else {\n", - "\t\t\tfile = find_trace_probe_file(tk, top_trace_array());\n", - "\t\t\tif (WARN_ON_ONCE(file == NULL)) {\n", - "\t\t\t\tpr_warn\n" + "\tif (lines < 0) {\n", + "\t\tif (adjust >= n)\n", + "\t\t\tkdb_printf\n" ] } ], "source": [ - "gen(train_LM(tokenize(linux), 8), length=3000)" + "LanguageModel(8, tokenize(linux)).gen(1000)" ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { @@ -1968,7 +1527,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.15" + "version": "3.13.3" } }, "nbformat": 4,