# moar-v3

a small language model pretrained from scratch by lurk, on nothing but what its lurkers read.

## what it is
- decoder-only transformer, **5,310,720 parameters**, trained from random init (no pretrained weights, no distillation, no outside text).
- 4 layers, d_model 256, 4 heads, context 256 tokens; rmsnorm, rotary positions, swiglu mlp, tied input/output embeddings, no biases.
- tokenizer: byte-level bpe, 8,192 tokens, trained on the same crawl. `<|doc|>` (id 0) separates documents, `<|pad|>` (id 1) is unused padding.

## data
- the lurk crawl only: 7,323 accepted pages (page id <= 8609) from the crypto web — docs, whitepapers, research, wikis, forums — read by lurkers in real browsers.
- 17,426,665 tokens after tokenization; 98/2 train/val split by document.

## training
- 4,166 steps, 34,127,872 tokens seen, batch 8,192 tokens, peak lr 0.0015, warmup + cosine to 0.1x, adamw(0.9,0.95) wd=0.1 (matmuls only), grad clip 1.0, fp32.
- hardware: mps (Apple M1), 35.4 minutes.
- validation loss **3.505** nats/token (perplexity 33.3).

## limitations
- base model: it continues text. it does not chat, follow instructions, or know when it is wrong.
- tiny by modern standards and trained on a small, crypto-only crawl: expect fluent-ish nonsense, invented numbers, names and addresses.
- the crawl reflects whatever the crypto web says, including marketing and errors. not financial advice. never trust a contract address it writes.
- english only.

## files
- `model.safetensors` — weights (float32; `lm_head` is tied to `wte` and not stored)
- `config.json` — architecture + training config
- `tokenizer.json` — load with `tokenizers.Tokenizer.from_file`

## license
- weights, config and tokenizer: MIT, same as the lurk code.
- the training text belongs to its authors; it is not redistributed here. every source page is listed in the crawl (`/v1/pages`).

## samples (temperature 0.8, top-k 40)
- **a validator is** the Solana network designed to be a decentralized distributed system on one network. This article is also an update to how Bitcoin's consensus algorithm allows anyone to propose and vote by defining the block chain, as well as being a block for the purposes of this proposal. It helps help developers contribute towards better performance and decentralization through mining. Over the network, Bitcoin remains a new form of ledger that takes time between miners and mining and consensus changes. The Bitcoin network is a testament to its potential inefficiency and efficiency.
- **the whitepaper describes** Bitcoin Core’s consensus mechanism, called Balltree and Broaderutt Casper and Runtime-Symmetry called MASS. The consensus algorithm used for proof of concept, while an attack would make the protocol more secure and efficient as well as hardening transactions in a way that can be done by miners.
- **to bridge tokens** - The following is an easy-to-use interface that can be used with the new ERC20 address, as described by Github. - What should I do? - How to use the bridge token extension to transfer the new token (the destination side) from a wallet Trustless L2 token account which is not a payment address (i.e., this is a payment address that lets the payer sign sign or send it with an RPC endpoint). The standard API returns these tokens (address, RPC URL, etc.) directly.
- **proof of stake** A set of proof-of-Stake systems that are used for Proof-of-Stake consensus rules. However, this is not a way to do one or more (when known as leader votes exist) but this is no longer actively maintained. This system also allows the consensus code and its trust assumptions for each validator in the Proof-of-stake network, which can be used by multiple blockchains. If you are running an on-chain fork, however, you have the full set of proof-of-stake validators (i.e., the chain).
- **a liquidity pool** for the Solana network, including account indexers, RPC settings and configuration. The market continues to grow and matured by the ecosystem. The market is a stable platform that allows for efficient development of new features. This guide is expected to offer an overview of current-state channels, with the liquidity provisional fee mechanism set available here. What does Glider view the liquidity model for Solana? Game token swaps and rewards are rewarded by price discovery. They are used as collateral to earn yield on the SOL balance.
- **solana accounts** - Raydium Docs This page covers the Solana token program documentation for trading. What is Jupiter's transaction? Jupiter has built a comprehensive, user-friendly experience where users can integrate with their USDC in DeFi applications to find optimal markets on Solana. These include: - The SOL (SOL) on Solana: SPL ETH (USDC), staking, and token supply—whether from the wallet’s SOL price and native stake accounts, you will need to be able to earn rewards.