Re: [RFC PATCH] thermal: add generic cpu hotplug cooling device

[Zoran Markovic]

Any comments on this proposed feature and implementation? Apparently
it's also useful for server systems.
Thanks,
Zoran

On 20 September 2013 15:15, Zoran Markovic <zoran.markovic@xxxxxxxxxx> wrote:
> This patch implements a generic CPU hotplug cooling device. The
> implementation scales down the number of running CPUs when temperature
> increases through a thermal trip point and prevents booting CPUs
> until thermal conditions are restored. Upon restoration, the action
> of starting up a CPU is left to another entity (e.g. CPU offline
> governor, for which a patch is in the works).
>
> In the past two years, ARM considerably reduced the time required for
> CPUs to boot and shutdown; this time is now measured in microseconds.
> This patch is predominantly intended for ARM big.LITTLE architectures
> where big cores are expected to have a much bigger impact on thermal
> budget than little cores, resulting in fast temperature ramps to a trip
> point, i.e. thermal runaways. Switching off the big core(s) may be one
> of the recovery mechanisms to restore system temperature, but the actual
> strategy is left to the thermal governor.
>
> The assumption is that CPU shutdown/startup is a rare event, so no
> attempt was made to make the code atomic, i.e. the code evidently races
> with CPU hotplug driver. The set_cur_state() function offlines CPUs
> iteratively one at a time, checking the cooling state before each CPU
> shutdown. A hotplug notifier callback validates any CPU boot requests
> against current cooling state and approves/denies accordingly. This
> mechanism guarantees that the desired cooling state could be reached in a
> maximum of d-c iterations, where d and c are the "desired" and "current"
> cooling states expressed in the number of offline CPUs.
>
> Credits to Amit Daniel Kachhap for initial attempt to upstream this feature.
>
> Cc: Zhang Rui <rui.zhang@xxxxxxxxx>
> Cc: Eduardo Valentin <eduardo.valentin@xxxxxx>
> Cc: Rob Landley <rob@xxxxxxxxxxx>
> Cc: Amit Daniel Kachhap <amit.daniel@xxxxxxxxxxx>
> Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
> Cc: Durgadoss R <durgadoss.r@xxxxxxxxx>
> Cc: Christian Daudt <bcm@xxxxxxxxxxxxx>
> Cc: James King <james.king@xxxxxxxxxx>
> Signed-off-by: Zoran Markovic <zoran.markovic@xxxxxxxxxx>
> ---
>  Documentation/thermal/cpu-cooling-api.txt |   17 ++
>  drivers/thermal/Kconfig                   |   10 +
>  drivers/thermal/Makefile                  |    1 +
>  drivers/thermal/cpu_hotplug.c             |  362 +++++++++++++++++++++++++++++
>  include/linux/cpuhp_cooling.h             |   57 +++++
>  5 files changed, 447 insertions(+)
>  create mode 100644 drivers/thermal/cpu_hotplug.c
>  create mode 100644 include/linux/cpuhp_cooling.h
>
> diff --git a/Documentation/thermal/cpu-cooling-api.txt b/Documentation/thermal/cpu-cooling-api.txt
> index fca24c9..2f94f68 100644
> --- a/Documentation/thermal/cpu-cooling-api.txt
> +++ b/Documentation/thermal/cpu-cooling-api.txt
> @@ -30,3 +30,20 @@ the user. The registration APIs returns the cooling device pointer.
>      This interface function unregisters the "thermal-cpufreq-%x" cooling device.
>
>      cdev: Cooling device pointer which has to be unregistered.
> +
> +1.2 cpu hotplug registration/unregistration APIs
> +1.2.1 struct thermal_cooling_device *cpuhp_cooling_register(
> +       struct cpumask *cpus, const char *ext)
> +
> +    This function creates and registers a cpu hotplug cooling device with
> +    the name "cpu-hotplug-%s".
> +
> +    cpus: cpumask of cpu cores participating in cooling.
> +    ext: instance-specific name of device
> +
> +1.2.2 void cpuhotplug_cooling_unregister(struct thermal_cooling_device *cdev)
> +
> +    This function unregisters and frees the cpu hotplug cooling device cdev.
> +
> +    cdev: Pointer to cooling device to unregister.
> +
> diff --git a/drivers/thermal/Kconfig b/drivers/thermal/Kconfig
> index 52b6ed7..3509100 100644
> --- a/drivers/thermal/Kconfig
> +++ b/drivers/thermal/Kconfig
> @@ -79,6 +79,16 @@ config CPU_THERMAL
>
>           If you want this support, you should say Y here.
>
> +config CPU_THERMAL_HOTPLUG
> +       bool "Generic CPU hotplug cooling"
> +       depends on HOTPLUG_CPU
> +       help
> +         Shutdown CPUs to prevent the device from overheating. This feature
> +         uses generic CPU hot-unplug capabilities to control device
> +         temperature. When the temperature increases over a trip point, a
> +         random subset of CPUs is shut down to reach the desired cooling
> +         state.
> +
>  config THERMAL_EMULATION
>         bool "Thermal emulation mode support"
>         help
> diff --git a/drivers/thermal/Makefile b/drivers/thermal/Makefile
> index 5ee0db0..0bd08be 100644
> --- a/drivers/thermal/Makefile
> +++ b/drivers/thermal/Makefile
> @@ -12,6 +12,7 @@ thermal_sys-$(CONFIG_THERMAL_GOV_USER_SPACE)  += user_space.o
>
>  # cpufreq cooling
>  thermal_sys-$(CONFIG_CPU_THERMAL)      += cpu_cooling.o
> +thermal_sys-$(CONFIG_CPU_THERMAL_HOTPLUG)      += cpu_hotplug.o
>
>  # platform thermal drivers
>  obj-$(CONFIG_SPEAR_THERMAL)    += spear_thermal.o
> diff --git a/drivers/thermal/cpu_hotplug.c b/drivers/thermal/cpu_hotplug.c
> new file mode 100644
> index 0000000..8c3021e
> --- /dev/null
> +++ b/drivers/thermal/cpu_hotplug.c
> @@ -0,0 +1,362 @@
> +/*
> + *  drivers/thermal/cpu_hotplug.c
> + *
> + *  Copyright (C) 2013  Broadcom Corporation Ltd.
> + *  Copyright (C) 2013  Zoran Markovic <zoran.markovic@xxxxxxxxxx>
> + *
> + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> + *  This program is free software; you can redistribute it and/or modify
> + *  it under the terms of the GNU General Public License as published by
> + *  the Free Software Foundation; version 2 of the License.
> + *
> + *  This program is distributed in the hope that it will be useful, but
> + *  WITHOUT ANY WARRANTY; without even the implied warranty of
> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + *  General Public License for more details.
> + *
> + *  You should have received a copy of the GNU General Public License along
> + *  with this program; if not, write to the Free Software Foundation, Inc.,
> + *  59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
> + *
> + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> + */
> +#include <linux/module.h>
> +#include <linux/thermal.h>
> +#include <linux/workqueue.h>
> +#include <linux/cpu.h>
> +#include <linux/err.h>
> +#include <linux/slab.h>
> +#include <linux/cpuhp_cooling.h>
> +
> +/**
> + * struct cpuhotplug_cooling_device - cpu hotplug cooling device data
> + * @cpus: cpu mask representing cpus that can be hot-unplugged for cooling
> + * @cdev: pointer to generic cooling device
> + */
> +struct cpuhotplug_cooling_device {
> +       unsigned int target;
> +       struct cpumask cpus;
> +       struct thermal_cooling_device *cdev;
> +       struct list_head list;
> +};
> +
> +/**
> + * cpuhotplug_list - list of all cpu hotplug cooling devices. Traversed
> + * by cpu hotplug notifier to check constraints on booting cpus. Locked
> + * by cpuhotplug_cooling_lock mutex.
> + */
> +static LIST_HEAD(cpuhotplug_list);
> +static DEFINE_MUTEX(cpuhotplug_cooling_lock);
> +
> +/**
> + * boot_cpu - return index of boot CPU; same criteria as in
> + * disable_nonboot_cpus()
> + */
> +static inline int boot_cpu(void)
> +{
> +       int cpu;
> +       get_online_cpus();
> +       cpu = cpumask_first(cpu_online_mask);
> +       put_online_cpus();
> +       return cpu;
> +}
> +
> +/**
> + * random_online_cpu - pick any online hot-unpluggable cpu
> + * @d: pointer to cpuhotplug_cooling_device containing hot-pluggable cpu mask
> + */
> +static inline int random_online_cpu(struct cpuhotplug_cooling_device *d)
> +{
> +       int cpu;
> +
> +       get_online_cpus();
> +       cpu = any_online_cpu(d->cpus);
> +       put_online_cpus();
> +
> +       return cpu;
> +}
> +
> +/**
> + * _num_offline_cpus - number of hot-pluggable cpus currently offline
> + * @d: pointer to cpuhotplug_cooling_device containing hot-pluggable cpu mask
> + */
> +static inline int _num_offline_cpus(struct cpuhotplug_cooling_device *d)
> +{
> +       struct cpumask offline;
> +
> +       cpumask_andnot(&offline, &(d->cpus), cpu_online_mask);
> +       return cpumask_weight(&offline);
> +}
> +
> +/**
> + * num_offline_cpus - same as _num_offline_cpus, but safe from background
> + * hotplug events.
> + * @d: pointer to cpuhotplug_cooling_device containing hot-pluggable cpu mask
> + */
> +static inline int num_offline_cpus(struct cpuhotplug_cooling_device *d)
> +{
> +       int num;
> +
> +       get_online_cpus();
> +       num = _num_offline_cpus(d);
> +       put_online_cpus();
> +
> +       return num;
> +}
> +
> +/**
> + * cpuhotplug_get_max_state - get maximum cooling state of device
> + * @cdev: pointer to generic cooling device
> + * @state: returned maximum cooling state
> + *
> + * Thermal framework callback to get the maximum cooling state of cpu
> + * hotplug cooling device.
> + *
> + * Return: always 0.
> + */
> +static int cpuhotplug_get_max_state(struct thermal_cooling_device *cdev,
> +                                   unsigned long *state)
> +{
> +       struct cpuhotplug_cooling_device *d = cdev->devdata;
> +
> +       /* defined as number of CPUs in hot-pluggable mask: this is invariant */
> +       *state = cpumask_weight(&(d->cpus));
> +
> +       return 0;
> +}
> +
> +/**
> + * cpuhotplug_get_cur_state - get current cooling state of device
> + * @cdev: pointer to generic cooling device
> + * @state: current cooling state
> + *
> + * Thermal framework callback to get the current cooling state of cpu
> + * hotplug cooling device.
> + *
> + * Return: always 0.
> + */
> +static int cpuhotplug_get_cur_state(struct thermal_cooling_device *cdev,
> +                                   unsigned long *state)
> +{
> +       struct cpuhotplug_cooling_device *d = cdev->devdata;
> +
> +       *state = d->target;
> +
> +       return 0;
> +}
> +
> +/**
> + * cpuhotplug_get_cur_state - set cooling state of device
> + * @cdev: pointer to generic cooling device
> + * @state: cooling state
> + *
> + * Thermal framework callback to set/change cooling state of cpu hotplug
> + * cooling device.
> + *
> + * Return: 0 on success, or error code otherwise
> + */
> +static int cpuhotplug_set_cur_state(struct thermal_cooling_device *cdev,
> +                                   unsigned long state)
> +{
> +       struct cpuhotplug_cooling_device *d = cdev->devdata;
> +       unsigned long cstate;
> +       unsigned int cpu;
> +       int err = 0;
> +
> +       if (state > cpumask_weight(&(d->cpus)))
> +               return -EINVAL; /* out of allowed range */
> +
> +       /*
> +        * Set target state here; hot-unplug CPUs if we are too hot, but
> +        * don't attempt to hot-plug CPUs if we're cold. Starting CPUs
> +        * should be left to CPUOffline governor.
> +        *
> +        * There is a chance that CPU hotplug driver is racing with this
> +        * code. Rather than trying to make the procedure atomic, iterate
> +        * until we reach the desired state, or signal error if the state
> +        * cannot be reached.
> +        *
> +        * Neither CPU hotplug nor this code is expected to run too often.
> +        */
> +       d->target = state;
> +
> +       /* compare desired cooling state to current cooling state */
> +       while ((cstate = num_offline_cpus(d)) < state && !err) {
> +               /* cstate < cstate: we're too hot, unplug any cpu */
> +               cpu = random_online_cpu(d);
> +               if (cpu < nr_cpu_ids)
> +                       err = work_on_cpu(boot_cpu(),
> +                                         (long(*)(void *))cpu_down,
> +                                         (void *)cpu);
> +                       /* on error, message would come from cpu_down() */
> +               else {
> +                       pr_warn("cpuhotplug: CPUs already down\n");
> +                       err = -EAGAIN;
> +               }
> +       }
> +
> +       return err;
> +}
> +
> +/* cpu hotplug cooling device ops */
> +static struct thermal_cooling_device_ops const cpuhotplug_cooling_ops = {
> +       .get_max_state = cpuhotplug_get_max_state,
> +       .get_cur_state = cpuhotplug_get_cur_state,
> +       .set_cur_state = cpuhotplug_set_cur_state,
> +};
> +
> +/**
> + * _cpu_startup_allowed - traverse list of hotplug cooling devices to
> + * check if startup of cpu violates thermal constraints
> + */
> +static inline int _cpu_startup_allowed(int cpu)
> +{
> +       struct cpuhotplug_cooling_device *d;
> +       int ret = 1;
> +
> +       /*
> +        * Prevent starting CPU if it violates any cooling
> +        * device's constraint. Called from hotplug notifier, so
> +        * cpu_up()/cpu_down() already holds a lock on hotplug
> +        * events.
> +        */
> +       mutex_lock(&cpuhotplug_cooling_lock);
> +       list_for_each_entry(d, &cpuhotplug_list, list) {
> +               if (cpumask_test_cpu(cpu, &(d->cpus)) &&
> +                   d->target >= _num_offline_cpus(d)) {
> +                       pr_warn("%s: CPU%d startup prevented\n",
> +                               dev_name(&(d->cdev->device)), cpu);
> +                       ret = 0;
> +                       break;
> +               }
> +       }
> +       mutex_unlock(&cpuhotplug_cooling_lock);
> +       return ret;
> +}
> +
> +/**
> + * cpuhotplug_thermal_notifier - notifier callback for CPU hotplug events.
> + * @nb: struct notifier_block
> + * @event: cpu hotplug event for which callback is invoked.
> + * @data: context data, in this particular case CPU index.
> + *
> + * Callback intercepting CPU hotplug events. Compares CPU hotplug action
> + * with current thermal state and allows/denies accordingly.
> + *
> + * Return: 0 (allow) or error (deny).
> + */
> +static int cpuhotplug_thermal_notifier(struct notifier_block *nb,
> +                                      unsigned long event, void *data)
> +{
> +       int cpu = (int)data;
> +
> +       switch (event) {
> +       case CPU_UP_PREPARE:
> +       case CPU_UP_PREPARE_FROZEN:
> +               /* _cpu_up() only cares about result from CPU_UP_PREPAREs */
> +               if (!_cpu_startup_allowed(cpu))
> +                       return notifier_from_errno(-EAGAIN);
> +               break;
> +       default:
> +               /* allow all other actions */
> +               break;
> +       }
> +       return 0;
> +}
> +
> +static struct notifier_block cpuhotplug_thermal_notifier_block = {
> +       .notifier_call = cpuhotplug_thermal_notifier,
> +};
> +
> +/**
> + * cpuhotplug_cooling_register - create cpu hotplug cooling device
> + * @cpus: cpumask of cpu cores participating in cooling
> + * @ext: instance-specific name of device
> + *
> + * Creates and registers a cpu hotplug cooling device with the name
> + * "cpu-hotplug-<ext>".
> + *
> + * Return: valid pointer to cpuhotplug_cooling_device struct on success,
> + * corresponding ERR_PTR() on failure.
> + */
> +struct thermal_cooling_device *
> +cpuhotplug_cooling_register(const struct cpumask *cpus, const char *ext)
> +{
> +       struct thermal_cooling_device *cdev;
> +       struct cpuhotplug_cooling_device *cpuhotplug_cdev;
> +       struct cpumask test;
> +       char name[THERMAL_NAME_LENGTH];
> +       int err;
> +
> +       /* test if we passed in a good cpumask */
> +       cpu_maps_update_begin();
> +       cpumask_and(&test, cpus, cpu_possible_mask);
> +       cpu_maps_update_done();
> +
> +       if (cpumask_test_cpu(boot_cpu(), &test)) {
> +               pr_warn("cannot hot-plug boot CPU%d\n", boot_cpu());
> +               cpumask_clear_cpu(boot_cpu(), &test);
> +       }
> +       if (cpumask_empty(&test)) {
> +               pr_err("CPUs unavailable for hot-plug cooling\n");
> +               err = -EINVAL;
> +               goto out;
> +       }
> +
> +       cpuhotplug_cdev = kzalloc(sizeof(struct cpuhotplug_cooling_device),
> +                            GFP_KERNEL);
> +       if (!cpuhotplug_cdev) {
> +               err = -ENOMEM;
> +               goto out;
> +       }
> +
> +       cpumask_copy(&cpuhotplug_cdev->cpus, &test);
> +
> +       snprintf(name, sizeof(name), "cpu-hotplug-%s", ext);
> +
> +       cpuhotplug_cdev->target = 0;
> +       cdev = thermal_cooling_device_register(name, cpuhotplug_cdev,
> +                                              &cpuhotplug_cooling_ops);
> +       if (!cdev) {
> +               pr_err("%s: cooling device registration failed.\n", name);
> +               err = -EINVAL;
> +               goto out_free;
> +       }
> +       cpuhotplug_cdev->cdev = cdev;
> +
> +       mutex_lock(&cpuhotplug_cooling_lock);
> +       if (list_empty(&cpuhotplug_list))
> +               register_cpu_notifier(&cpuhotplug_thermal_notifier_block);
> +       list_add(&(cpuhotplug_cdev->list), &cpuhotplug_list);
> +       mutex_unlock(&cpuhotplug_cooling_lock);
> +
> +       return cdev;
> +
> +out_free:
> +       kfree(cpuhotplug_cdev);
> +out:
> +       return ERR_PTR(err);
> +}
> +EXPORT_SYMBOL_GPL(cpuhotplug_cooling_register);
> +
> +/**
> + * cpuhotplug_cooling_unregister - remove cpu hotplug cooling device
> + * @cdev: cooling device to remove
> + *
> + * Unregisters and frees the cpu hotplug cooling device.
> + */
> +void cpuhotplug_cooling_unregister(struct thermal_cooling_device *cdev)
> +{
> +       struct cpuhotplug_cooling_device *cpuhotplug_cdev = cdev->devdata;
> +
> +       mutex_lock(&cpuhotplug_cooling_lock);
> +       list_del(&(cpuhotplug_cdev->list));
> +       if (list_empty(&cpuhotplug_list))
> +               unregister_cpu_notifier(&cpuhotplug_thermal_notifier_block);
> +       mutex_unlock(&cpuhotplug_cooling_lock);
> +
> +       thermal_cooling_device_unregister(cpuhotplug_cdev->cdev);
> +       kfree(cpuhotplug_cdev);
> +}
> +EXPORT_SYMBOL_GPL(cpuhotplug_cooling_unregister);
> +
> diff --git a/include/linux/cpuhp_cooling.h b/include/linux/cpuhp_cooling.h
> new file mode 100644
> index 0000000..ace1d5b
> --- /dev/null
> +++ b/include/linux/cpuhp_cooling.h
> @@ -0,0 +1,57 @@
> +/*
> + *  linux/include/linux/cpuhp_cooling.h
> + *
> + *  Copyright (C) 2013 Broadcom Corporation Ltd.
> + *  Copyright (C) 2013 Zoran Markovic <zoran.markovic@xxxxxxxxxx>
> + *
> + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> + *  This program is free software; you can redistribute it and/or modify
> + *  it under the terms of the GNU General Public License as published by
> + *  the Free Software Foundation; version 2 of the License.
> + *
> + *  This program is distributed in the hope that it will be useful, but
> + *  WITHOUT ANY WARRANTY; without even the implied warranty of
> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + *  General Public License for more details.
> + *
> + *  You should have received a copy of the GNU General Public License along
> + *  with this program; if not, write to the Free Software Foundation, Inc.,
> + *  59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
> + *
> + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> + */
> +
> +#ifndef __CPUHP_COOLING_H__
> +#define __CPUHP_COOLING_H__
> +
> +#include <linux/thermal.h>
> +#include <linux/cpumask.h>
> +
> +#ifdef CONFIG_CPU_THERMAL_HOTPLUG
> +/**
> + * cpuhotplug_cooling_register - create cpu hotplug cooling device.
> + * @cpus: cpumask of hot-pluggable cpus
> + * @ext: instance-specific device name
> + */
> +struct thermal_cooling_device *
> +cpuhotplug_cooling_register(const struct cpumask *cpus, const char *ext);
> +
> +/**
> + * cpuhotplug_cooling_unregister - remove cpu hoptlug cooling device.
> + * @cdev: thermal cooling device pointer.
> + */
> +void cpuhotplug_cooling_unregister(struct thermal_cooling_device *cdev);
> +#else /* !CONFIG_CPU_THERMAL_HOTPLUG */
> +static inline struct thermal_cooling_device *
> +cpuhotplug_cooling_register(const struct cpumask *cpus, const char *ext)
> +{
> +       return NULL;
> +}
> +static inline
> +void cpuhotplug_cooling_unregister(struct thermal_cooling_device *cdev)
> +{
> +       return;
> +}
> +#endif /* CONFIG_CPU_THERMAL_HOTPLUG */
> +
> +#endif /* __CPU_COOLING_H__ */
> --
> 1.7.9.5
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/