Aw: Re: Re: new module to check constant memory for corruption

From: Alexander . Kleinsorge
Date: Sun Apr 13 2014 - 13:21:13 EST

1. KERN_EMERG is now KERN_INFO or ALERT (depending on ftrace_enabled).
2. there is no trigger for real ram errors possible! (only for out of bound writes)
Therefore it is checked periodically and can also be triggered by /proc/ramcheck

Alex

ÂOn Sun, 13 Apr 2014 03:33:10 +0200, Alexander.Kleinsorge [at] gmx said:

> printk(KERN_EMERG MODNAME "error: const kernel memory is broken (%08lx != 0), please reboot!", g_SumLast);

Make a list of all the things that can dynamically modify kernel text while
it's running, starting with the alternatives code, and adding in the various
tracing and debugging tools, and ask yourself if KERN_EMERG is the right
choice here...

Also, that printk is in the wrong place - it only fires if somebody reads
the /proc file, and it *should* trigger as soon as practical after a
problem has been detected.

Bonus points for computing what percent of single, double, triple, and
other error syndromes your xor detects (hint - what is the behavior of a
dead row or column of bits?).
Â

Gesendet:ÂSonntag, 13. April 2014 um 12:26 Uhr
Von:Â"Richard Weinberger" <richard.weinberger@xxxxxxxxx>
An:ÂAlexander.Kleinsorge@xxxxxx
Cc:Â"Andi Kleen" <andi@xxxxxxxxxxxxxx>, LKML <linux-kernel@xxxxxxxxxxxxxxx>
Betreff:ÂRe: Re: new module to check constant memory for corruption
On Sun, Apr 13, 2014 at 12:14 PM, <Alexander.Kleinsorge@xxxxxx> wrote:
> Hi Andi,
>
> the module considers only the adress range between: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata").
> this range has a typical size of 10..20 mb (depending on kernel-version and arch).
> see files: linux-3.*\arch\x86\mm\init_32.c + init_64.c
> function: void mark_rodata_ro(void)
> "Write protecting the kernel text: %luk\n"
> "Write protecting the kernel read-only data: %luk\n"
> dmesg | grep protecting
>
> your question: there are no writes in this write protected adress range (e.g. kernel code).

And what happens if one enables dynamic ftrace or other kernel
features which modify kernel code?

> my idea is to calculate a checksum (xor is fastest) over this range and check later (periodically) if its unchanged.
> see source code download (5 KB): http://tauruz.homeip.net/ramcheck.tgz
> the code is working fine and the checksum is (as expected) constant (at least for many hours).
>
> regards, Alexander
>
>
> Gesendet: Sonntag, 13. April 2014 um 05:00 Uhr
> Von: "Andi Kleen" <andi@xxxxxxxxxxxxxx>
> An: Alexander.Kleinsorge@xxxxxx
> Cc: linux-kernel@xxxxxxxxxxxxxxx
> Betreff: Re: new module to check constant memory for corruption
> Alexander.Kleinsorge@xxxxxx writes:
>
>> ramcheck kernel module
>> new module to check constant memory for corruption
>>
>> detect corruption of constant kernel memory (text and data) periodically.
>> runtime costs about 1..2 ms per sec (about 10 mb with 5 mb/ms),
>> which is distributed over 8 (BLOCKS) time partitions (less than half
>> ms per sec).
>> in case of checksum (xor) error, an kernel log is posted.
>> manual trigger via /proc/ramcheck is possible.
>> range: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata")
>
>
> Can you explain how this works? How does it handle legal writes?
>
> If it just checks its own memory it could be done in user space.
>
> -Andi
>
> --
> ak@xxxxxxxxxxxxxxx -- Speaking for myself only
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html[http://vger.kernel.org/majordomo-info.html]
> Please read the FAQ at http://www.tux.org/lkml/[http://www.tux.org/lkml/]



--
Thanks,
//richard/* ramcheck.c - checks predefined kernel memory region
*
* KernelRamCheck by
*
* Copyright 2014 Alexander Kleinsorge <alexander[dot]kleinsorge[at]gmx[dot]de>
* Copyright 2014 Benjamin Schroedl <benjamin[at]dev-tec[dot]de>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*
*/

// Defining __KERNEL__ and MODULE allows us to access kernel-level code not usually available to userspace programs.
/*
#undef __KERNEL__
#define __KERNEL__

#undef MODULE
#define MODULE
*/
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kallsyms.h>
#include <linux/kernel.h>
#include <linux/pfn.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <linux/timer.h>
#include <linux/delay.h>
#include <crypto/algapi.h>
#include <asm/uaccess.h>
#include <asm/sections.h>
#include <linux/time.h>
#include <linux/ftrace.h>
#include <asm/cacheflush.h>
/* read file: /proc/sys/kernel/ftrace_enabled */
#include <linux/fs.h>
#include <asm/segment.h>
#include <asm/uaccess.h>
#include <linux/buffer_head.h>

#ifdef CONFIG_DYNAMIC_FTRACE
# warning "RAMCHECK: using FTRACE can cause false alarm."
/* cat /proc/sys/kernel/ftrace_enabled (default 1) */
#endif

#define DRIVER_AUTHOR "Alexander Kleinsorge <alexander[dot]kleinsorge[at]gmx[dot]de> and Benjamin Schroedl <benjamin[at]dev-tec[dot]de>"
#define DRIVER_DESC "checks predefined memory kernel region"
#define proc_fs_name "ramcheck"
#define MODNAME proc_fs_name" "

#define CHECK_TIMER_MS 1000
#define BLOCK ((sizeof(long) <= 4) ? 1u << 20 : 2u << 20) /* 64bit CPU is often faster (bigger blocks at once) */
#define BLOCKS 8

MODULE_LICENSE("GPL");
MODULE_AUTHOR(DRIVER_AUTHOR);
MODULE_DESCRIPTION(DRIVER_DESC);


static void ram_check_exit(void);
static int ram_check_init(void);
static int ram_check_print(struct seq_file *m, void *v);
static int ram_check_run(void);
void ram_check_timer_callback(unsigned long data);
static int ram_check_open(struct inode *inode, struct file *file);

static struct timer_list ram_check_timer;
struct proc_dir_entry *ram_check_proc_file;
struct file_operations proc_fops = {
.open = ram_check_open,
.read = seq_read,
.llseek = seq_lseek,
.release = single_release,
};


/* Global Variables */

struct s_ramcheck_init
{
unsigned long RangeSize;
unsigned long RangeStart;
unsigned long SumFirst;
unsigned long BlockSum[BLOCKS];
unsigned int TimerActive;
unsigned int dummy; // allign 64 bit
};
static struct s_ramcheck_init g_init = {0u,0u,0u, {0},0u,0u};

static unsigned long g_SumLast = 0;
static unsigned int g_TempCount = 0;
static unsigned int g_Timers = 0;


/* Functions */

/* simple checksum/hash, xor is fastest (~1-6 MB/ms) */
static unsigned long calcChecksumXor(const void* ptr, const unsigned long bytes)
{
unsigned long ret, i;
const unsigned long long* ptr64 = (const unsigned long long*) ptr;
const unsigned long len = bytes / sizeof(*ptr64);
unsigned long long sum64 = 0u;

for (i=0; i < len; i++) {
sum64 ^= ptr64[i];
}

/* return 32 or 64 bit possible */
ret = ((unsigned long) sum64) ^ ((unsigned long) (sum64>>32));

return ret;
}

/* merge 4 to 1 byte ==> less info (security reason) */
static unsigned char shrink2byte(unsigned long u)
{
unsigned long ret = (u) ^ (u>>8) ^ (u>>16) ^ (u>>24);
/* (lowest) 32 bit --> 8 bit */
return (unsigned char) ret;
}

static unsigned long long getTimeUS(void)
{
struct timespec ts;
ktime_get_ts( &ts );

return (ts.tv_sec*USEC_PER_SEC) + (ts.tv_nsec/1000u);
}

// check for const range (in case of strange checksum)
static int check_range(void)
{
unsigned long start = kallsyms_lookup_name("_text");
unsigned long size = kallsyms_lookup_name("__end_rodata") - start;
int ret = 0;

if (g_init.RangeSize && ((start != g_init.RangeStart) || (size != g_init.RangeSize))) {
printk(KERN_WARNING MODNAME "error: const kernel memory address changed (%08lx + %lu K), stop!\n", start, size/1024u);
g_init.RangeSize = 0; /* stop working */
ret = -1;
}

return ret;
}


static int fread_ftrace(void)
{
int rights = 0, flags = O_RDONLY;
const char path[64] = "/proc/sys/kernel/ftrace_enabled";
char data[4] = "\0\0\0";
struct file* filp = NULL;
mm_segment_t oldfs;
int err = 0, ret;
unsigned long long offset = 0u;

oldfs = get_fs();
set_fs(get_ds());
filp = filp_open(path, flags, rights);
set_fs(oldfs);
if (IS_ERR(filp)) {
err = PTR_ERR(filp);
return -1;
}

oldfs = get_fs();
set_fs(get_ds());
ret = vfs_read(filp, data, 1, &offset);
set_fs(oldfs);
filp_close(filp, NULL);

if (!ret) return -2;
if ('1' == data[0]) return 1;

return 0;
}

static int check_ftrace(void)
{
int ft = fread_ftrace();

#ifdef CONFIG_DYNAMIC_FTRACE
/* unknown: function_trace_stop, ftrace_enabled, ftrace_nr_registered_ops() */
if (ft > 0)
printk(KERN_WARNING MODNAME ": ftrace state = %d (enabled), false alarm possible!\n", ft);
else
printk(KERN_INFO MODNAME ": ftrace state = %d (disabled).\n", ft);
#else
if (ft >= 0)
printk(KERN_WARNING MODNAME ": CONFIG_DYNAMIC_FTRACE not defined, but useable (%d)!\n", ft);
#endif
return 0;
}

/* partitions to limit runtime to 1 ms (1..6 MB) */
static unsigned long check_block(unsigned long blk, unsigned long lastblk)
{
const unsigned long offset = blk * BLOCK;
unsigned long start = g_init.RangeStart + offset;
unsigned long size = BLOCK;
unsigned long ret;

if (blk >= lastblk) {
size = g_init.RangeSize % BLOCK;
if (blk > lastblk) {
return (~0u); /* should never happen */
}
}
ret = calcChecksumXor((const void *) start, size);

return ret;
}

/* INIT: call not before mark_rodata_ro(void) */
static int ram_check_init(void)
{
int ret;
unsigned long start, size, end;
unsigned long b, t, bps;

for (b=0; b<BLOCKS; b++) {
g_init.BlockSum[b] = 0u;
}

ram_check_proc_file = proc_create(proc_fs_name, 0, NULL, &proc_fops);

if (ram_check_proc_file == NULL) {
remove_proc_entry(proc_fs_name, NULL);
printk(KERN_ALERT MODNAME "error: Could not initialize /proc/%s\n", proc_fs_name);

return -ENOMEM;
}

start = kallsyms_lookup_name("_text");
end = kallsyms_lookup_name("__end_rodata");
/* kallsyms_lookup_name("__end_rodata_hpage_align"); ==> Bad Address! */
size = (end - start);

/* error handling on illegal size and ptr or size to big */
if ((start == 0) || (end == 0) || (end <= start) || (size > (1000u << 20))) {
printk(KERN_ALERT MODNAME "error: bad start or end adress [%lx + %lx]\n", start, size);

return -EFAULT;
}
if ((size < (1u << 20)) || (size > (200u << 20))) {
printk(KERN_INFO MODNAME "warning: strange const size = %lu MB\n", size>>20);
}

check_ftrace();

g_init.RangeStart = start;
g_init.RangeSize = size;

t = (unsigned long) getTimeUS();
for (b = 0; b <= size/BLOCK; b++) {
g_init.BlockSum[b % BLOCKS] ^= check_block(b, size/BLOCK);
}
t = (unsigned long) getTimeUS() - t; /* ca. 1k..7k Byte/us */
for (b = 0; b < BLOCKS; b++) {
g_init.SumFirst ^= g_init.BlockSum[b]; /* all together */
}

bps = (t > 0) ? (size / t) : 0;
printk(KERN_INFO MODNAME "init: %lu B/us, %lu kB, %d bit, (0x%02x, %u)\n",
bps, size>>10, (int)sizeof(g_init.SumFirst)*8, shrink2byte(g_init.SumFirst), BLOCKS);

/* TIMER init */
setup_timer(&ram_check_timer, ram_check_timer_callback, 0);

ret = mod_timer(&ram_check_timer, jiffies + msecs_to_jiffies(CHECK_TIMER_MS));
if (ret) {
printk(KERN_ALERT MODNAME "error: Failure in mod_timer\n");

return -EFAULT;
}
g_init.TimerActive = 1;
/* optional: clflush_cache_range or flush_icache_range (&g_init, &g_init + sizeof(g_init)); */

return 0;
}


static void ram_check_exit(void)
{
int ret;
g_init.RangeSize = 0; /* stop work */

remove_proc_entry(proc_fs_name, NULL);
check_range();

for (ret = 0; (g_init.TimerActive > 0) && (ret < 1000/20); ret ++) {
mdelay(20);
}

ret = del_timer_sync(&ram_check_timer);

if (ret) {
printk(KERN_WARNING MODNAME "warning: timer still in use on exit...(%d,%u)\n", ret, g_Timers);
} else {
printk(KERN_INFO MODNAME ": exit (%lx,%u)\n", g_SumLast, g_Timers);
}
}

/* file handler /proc/.. */
static int ram_check_print(struct seq_file *m, void *v)
{
int ret = 0;
unsigned long t = 0;

ret = check_range();
if (ret) {
seq_printf(m, "ERROR: const kernel memory address changed (%08lx), stop!\n", g_SumLast);

return -1;
}

if (g_SumLast != 0u) {
seq_printf(m, "ERROR: const kernel memory was earlier broken (%08lx), no check again!\n", g_SumLast);

return -2;
}

if (g_TempCount <= 2) { /* max 2 calls per timer (1 sec) */
unsigned long xor;
g_TempCount ++;
t = (unsigned long) getTimeUS();
xor = calcChecksumXor((const void *) g_init.RangeStart, g_init.RangeSize);
t = (unsigned long) getTimeUS() - t;
xor = g_init.SumFirst ^ xor;
if (xor) g_SumLast = xor; /* only write bad events */
} else {
g_TempCount ++;
seq_printf(m, "warning: too much triggers (%u) for ram_check (%08lx) !\n", g_TempCount, g_SumLast);
}

if (!g_SumLast) {
seq_printf(m, "const kernel memory is OK, size = %lu MB, t = %lu us (%u)\n", g_init.RangeSize>>20, t, g_Timers);
} else {
int ft = fread_ftrace();
if (ft > 0) {
printk(KERN_INFO MODNAME "warning: const kernel memory is broken (%08lx != 0), ftrace used (%d)?\n", g_SumLast, ft);
seq_printf(m, "WARNING: const kernel memory is broken (%08lx != 0), ftrace used (%d)?\n", g_SumLast, ft);
} else {
printk(KERN_ALERT MODNAME "error: const kernel memory is broken (%08lx != 0), please reboot!\n", g_SumLast);
seq_printf(m, "ERROR: const kernel memory is broken (%08lx != 0), please reboot!\n", g_SumLast);
}
ret = -3;
/* if (!FTRACE) : kernel panic (ram error) !! */
}

return ret;
}

/* only inside timer (1 sec) */
static int ram_check_run(void)
{
const unsigned int lastblk = (unsigned int) (g_init.RangeSize / BLOCK);
const unsigned int b = g_Timers % BLOCKS;
unsigned int l;
unsigned long x = 0;

if (0 == b) {
check_range(); /* check sometimes */
}

for (l = b; l <= lastblk; l += BLOCKS) {
x ^= check_block(l, lastblk);
}
x ^= g_init.BlockSum[b];

if (x != 0u) {
int ft = fread_ftrace();
check_range();
g_SumLast = x;
if (ft > 0) {
printk(KERN_ALERT MODNAME "error: const kernel memory broken (%08lx != 0), timer(%u, %u), ftrace=on(%d)!\n", x, g_Timers, b, ft);
/* kernel panic (ram error) !! */
} else {
printk(KERN_WARNING MODNAME "warning: const kernel memory broken (%08lx != 0), timer(%u, %u), ftrace=off(%d)!\n", x, g_Timers, b, ft);
}
}

return 0;
}

/* Timer Callback */
void ram_check_timer_callback(unsigned long data)
{
int ret;

g_Timers ++;
if (0 == g_init.RangeSize) {
g_init.TimerActive = 0;

return;
}

if (g_SumLast != 0u) { /* failure earlier detected, skip */
if (0 == (g_Timers % 32u)) {
printk(KERN_WARNING MODNAME "error: const kernel memory earlier broken (%08lx), skip !\n", g_SumLast);
}

return;
}

ram_check_run();

ret = mod_timer(&ram_check_timer, jiffies + msecs_to_jiffies(CHECK_TIMER_MS));
if (ret) {
printk(KERN_ALERT MODNAME "error: in mod_timer (%u, %d)\n", g_Timers, ret);
}

g_init.TimerActive = 1; /* is already set, but be safe */
g_TempCount = 0;
}


static int ram_check_open(struct inode *inode, struct file *file)
{
return single_open(file, ram_check_print, NULL);
}


module_init(ram_check_init);
module_exit(ram_check_exit);

Attachment: ramcheck.tgz
Description: Binary data